{"id":"f4e6c2b6-5752-4104-9afd-823d95042f91","arxiv_id":"2506.00875","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CC-Tuning fuses English feed-forward activations into non-English inputs during multilingual supervised fine-tuning, using a trainable Decision Maker and a least-squares Transform Matrix to simulate the connection at inference.","lead":"This paper introduces CC-Tuning, a way to fine-tune multilingual language models by blending internal English activation signals into non-English inputs during training. It reports gains over plain supervised fine-tuning on several multilingual understanding and generation benchmarks, at the cost of needing parallel English translations during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control for the source language/content of the fused activation leaves the core 'cross-lingual' mechanism unsupported; gains may stem from generic latent perturbation.","rationale":"The reader's weakest assumption concerns the Transform Matrix's linear approximation. That is a legitimate practical concern, but the paper already reports small |Delta| values between parallel-input inference and Transform-Matrix inference, so the concern is partly mitigated and its failure would weaken, not destroy, the central claim. The more fundamental and unaddressed issue is that the paper never isolates whether the auxiliary activation must be English-specific or even content-specific. The Decision Maker ablations and activation-type ablations hold the auxiliary language constant as English, so they cannot distinguish genuine cross-lingual transfer from generic latent perturbation or regularization. This directly targets the paper's central conceptual claim, not just a component. The empirical results remain plausible, and a single targeted control experiment would settle the issue, so the reader's CONDITIONAL verdict is preserved rather than upgraded or downgraded.","tokens_in":22918,"tokens_out":11020,"duration_ms":116302,"concrete_test":"Run the full CC-Tuning pipeline on LLaMA-3.1-8B with the original 20k Aya training data, but replace the auxiliary activation f_en_i,s in Eq. 5 with: (a) an FFN activation from a machine-translated version of xi into a third non-English language, e.g., French; (b) a random Gaussian vector matched in mean and variance to the English FFN activations; and (c) an FFN activation from an unrelated English sentence (content-mismatched). Evaluate on XNLI, MKQA, and XQuAD. If condition (b) or (c) matches or exceeds the English-activation condition, the cross-lingual specificity claim fails; if English clearly wins, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CC-Tuning establishes a cross-lingual connection mechanism at the latent level. The training intervention in Eq. 5 is tilde f_i,1 = f_i,1 + f_en_i,s, where f_en_i,s is the Decision-Maker-selected FFN activation from the English translation of the same input. The paper never varies the source of this auxiliary activation. The ablations in Figure 3 vary only the pooling strategy over English layers (Decision Maker vs. Mean/Random), and Figure 4 varies the activation type (FFN vs. attention vs. block), always keeping the auxiliary input as English. Thus the observed improvements over vanilla SFT could be explained by (a) the cross-lingual content of English activations, (b) any content-aligned auxiliary activation, e.g., a translation into another language, or (c) a regularization effect of injecting a large, content-dependent latent vector into the first-layer residual stream. The +EN degradation is not a clean control because it also changes the training data distribution to more than 50% English. Without a control that replaces f_en_i,s with a same-language mismatched activation or a matched random vector, the headline claim that the mechanism is specifically cross-lingual is not established. This is load-bearing because the paper's novelty and the 'latent-level alternative to data-level augmentation' framing depend on English linguistic information being transferred, not on generic perturbation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CC-Tuning, a multilingual supervised fine-tuning method that operates on latent feed-forward activations. For each non-English training example, an English translation is also passed through the model; a trainable Decision Maker selects one layer's feed-forward activation, and that activation is added to the first-layer feed-forward output of the non-English input before response generation (Eqs. 3-5). The training objective is unchanged from standard SFT. At inference, a Transform Matrix W_T, fit by least squares on 1,000 parallel pairs (Eqs. 7-8), maps non-English feed-forward activations into English-like activations, so that parallel inputs are not needed. The paper evaluates on six benchmarks covering 22 languages, comparing CC-Tuning with vanilla multilingual SFT and with data-augmentation baselines (+EN, +MT, +SDRRL) on LLaMA-3.1-8B and Qwen2.5-7B. The reported results show larger gains for LLaMA-3.1-8B and mixed or negligible gains for Qwen2.5-7B.","tokens_in":23178,"tokens_out":7988,"duration_ms":68200,"significance":"The work is significant if the mechanism is confirmed: it proposes an explicit latent-level alternative to data-level augmentation, requires no additional training loss, adds negligible parameters, and includes a practical inference-time procedure. The experimental coverage is broad, and the design is motivated by prior evidence that English feed-forward activations can improve non-English performance. However, the current evidence is not yet convincing at the claimed level. The main results lack multiple seeds or error bars, the Transform Matrix is validated in-sample on its own fitting objective, and the ablations do not isolate English-specific content from generic auxiliary or regularizing signals. These are empirical gaps that can be addressed in revision; the conceptual proposal remains interesting.","major_comments":[{"comment":"The claim that CC-Tuning 'significantly outperforms' vanilla SFT is not supported by the reported numbers. For Qwen2.5-7B, MMMLU decreases by 0.40 points and MKQA, XQuAD, and XLSum improve by only +0.11, +0.11, and +0.10 points, respectively; these differences are within the typical run-to-run variance of 7B-scale instruction tuning. No standard deviations, confidence intervals, or multiple seeds are reported for any result. The LLaMA-3.1-8B results are more favorable (e.g., XNLI +6.54, XStoryCloze +5.37), but even those lack variance estimates. I ask for at least three seeds with means and standard deviations for the main CC-Tuning vs. ML-SFT comparison, or a paired significance test across languages, before the superiority claim can be accepted.","section":"Table 1, Section 4.2"},{"comment":"The MSE used to validate the Transform Matrix is computed on the same 1,000 parallel pairs used to fit W_T. Since W_T is the least-squares minimizer of exactly that objective, a low in-sample MSE is expected by construction and does not show that the mapping transfers to new inputs or languages. The |Δ| results compare parallel-bilingual inference with Transform-Matrix inference in the same in-sample setting. Please supplement with hold-out or cross-validated MSE and downstream results on examples not used to fit W_T. A per-language or per-layer breakdown would also help assess the single-linear-matrix assumption.","section":"Section 4.3(1), Eqs. (7)-(8), Table 2"},{"comment":"No condition varies the source of the injected activation, so the specific 'cross-lingual' attribution is not established. The auxiliary vector is always the English feed-forward activation from the parallel translation of the same input, and the ablations only vary the pooling strategy and the activation type while keeping English as the source. The observed gains could therefore come from content-aligned auxiliary information, from any language's translation, or from a regularizing effect of injecting a task-relevant vector into the first residual stream. The +EN condition in Table 1 is not a clean control because it also changes the training data so that more than half is English. I recommend controls that replace f_en_{i,s} with: (i) a same-language activation from a different example, (ii) a norm-matched random vector, and (iii) a third-language translation of the same input.","section":"Section 3.2.1, Eq. (5), Figures 3-4"}],"minor_comments":[{"comment":"The summation index in the second term is written 'LX i=l'; this appears to be a typo for L and l=1, and the first term has a similar issue. Please correct the limits.","section":"Eq. (8)"},{"comment":"The language header for XQuAD (en, ar, bn, es, hi, id, ko, pt, sw, yo) is inconsistent with the language list given in Appendix A.2.1 for XQuAD (en, ar, de, el, hi, ru, th, tr, vi, zh). Please correct the table header and verify the per-language values.","section":"Table 8"},{"comment":"The t-SNE discussion is purely qualitative. Please add a quantitative measure of clustering or alignment, or clarify what 'more compact' is relative to; otherwise, avoid drawing strong conclusions from the visualization.","section":"Figure 7"},{"comment":"The code link is given as 'CC-Tuning' without a URL; a revision should provide the actual repository address.","section":"Abstract"},{"comment":"The Transform Matrix is called 'training-free' even though it is fit on 1,000 parallel pairs. Consider calling it 'without gradient-based training' or 'post-hoc' to avoid confusion.","section":"Section 3.2.2"},{"comment":"Please specify the Gumbel-Softmax temperature and whether hard or soft samples are used during training and inference; this affects the interpretation of the Decision Maker's layer selection.","section":"Eqs. (3)-(4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope. The proposal is simple and well-motivated, and the LLaMA results are promising, so I do not recommend rejection. The main gaps are evidentiary: variance reporting, out-of-sample validation of W_T, and controls for the source of the injected activation. I would also flag that the code link appears to be a placeholder, which should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read before deciding what to do with it. The core idea is new and simple: during multilingual SFT, take the FFN activations from the English translation of the training input, pick a layer with a learnable Gumbel-Softmax gate (the Decision Maker), and add that activation into the first-layer residual stream of the non-English input. At inference you don't have the parallel text, so they fit one linear transform to approximate English activations from non-English activations, and use that to simulate the connection. That's a clean, low-overhead intervention: around 1.15x training time, negligible extra parameters, and it works on both NLU and generation tasks across 22 languages.\n\nWhat the paper does well: it takes the prior observation that English FFN activations can be transplanted cross-lingually (Ye et al. 2024b) and turns it into a practical training procedure with a sensible inference-time approximation. The ablations are informative — Decision Maker helps over mean/random pooling, FFN activations beat attention/block activations, and the layer-distribution analysis is a nice diagnostic. On LLaMA-3.1-8B the gains look solid: +6.54 XNLI, +5.37 XStoryCloze, +1.21 XQuAD. On Qwen2.5-7B the gains are small and sometimes negative; the paper reports them honestly rather than cherry-picking.\n\nSoft spots, in rough order of importance. First, no error bars or multiple seeds, and some Qwen deltas are a fraction of a point, so I can't tell which of those are real. Second, the Transform Matrix's MSE is computed on the same data used to fit it, which is circular; the downstream |Δ| numbers in Table 2 are the meaningful check and those look fine. Third — and this is the one I'd want a referee to push on — nothing in the paper isolates the cross-lingual content of the auxiliary activation. The mechanism is called cross-lingual, but the ablations always use English activations as the auxiliary input. A same-language mismatched activation or a content-matched random vector would tell you whether the gains come from English linguistic information or from a generic latent perturbation that regularizes training. The +EN degradation is not a clean control because it changes the data distribution. This is a real hole in the mechanistic story, but the headline empirical claim — CC-Tuning beats vanilla SFT — would survive even if the effect turns out to be generic latent augmentation.\n\nWho's it for: anyone working on multilingual instruction tuning or latent-space interventions. It deserves a serious referee; I'd send it out, with the missing control and variance estimates as requested revisions.","headline":"A practical latent-fusion trick for multilingual SFT with real but uneven gains; the cross-lingual mechanism itself isn't isolated, so treat the causal claim with caution.","tokens_in":23808,"tokens_out":3494,"would_cite":true,"duration_ms":30882,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CC-Tuning claims that explicitly fusing English feed-forward activations into non-English forward passes during supervised fine-tuning improves multilingual performance beyond vanilla SFT, serving as a latent-level alternative to…","keywords":["multilingual supervised fine-tuning","cross-lingual latent interaction","feed-forward activations","Decision Maker","Transform Matrix","multilingual LLM","data augmentation alternative","curse of multilinguality"],"falsifier":"A decisive test is to evaluate the Transform Matrix's per-language residual $\\|f_{i,l}W_T - f^{\\mathrm{en}}_{i,l}\\|_2$ for languages not among the 1,000 parallel pairs used to fit it. If typologically distant or low-resource languages show residuals one order of magnitude above the average, the single linear map assumption fails and the inference-time simulation is not genuinely cross-lingual. A second check is to compare CC-Tuning's result using true English parallel activations against its result using $W_T$ language-by-language: the paper reports small average $|\\Delta|$, but the claim would be falsified if the gap grows systematically with language distance.","tokens_in":22683,"feed_emoji":"🌐","tokens_out":5755,"duration_ms":49308,"temperature":0.7,"pith_summary":"CC-Tuning is a multilingual fine-tuning method that improves non-English performance by explicitly connecting English and non-English representations inside the model, rather than by adding translated training text. During training it runs each non-English input alongside its English translation, extracts feed-forward activations from the English pass, selects the most useful layer with a trainable Decision Maker, and adds that activation into the first feed-forward layer of the non-English pass. The loss function is unchanged, and the added parameters are negligible. At inference the method needs no parallel input: a Transform Matrix learned from 1,000 parallel pairs maps non-English activations into English space to reproduce the connection. Across six benchmarks and 22 languages on two 7–8B models, the paper reports that CC-Tuning outperforms vanilla supervised fine-tuning and is competitive with or better than data-level augmentation and distillation baselines.","feed_headline":"Fusing English hidden states lifts multilingual fine-tuning","feed_subtitle":"A latent-level fusion of feed-forward activations outperforms data augmentation across six benchmarks and 22 languages.","key_machinery":"The central object is the cross-lingual connection mechanism applied to feed-forward activations just before the response start token. It has three parts: (1) an Adaptive Decision Maker, a trainable linear layer mapping combined English activations and the non-English embedding to a distribution over layers, with Gumbel-Softmax selecting a beneficial layer $s$; (2) a Latent Feed Forward Connection, the injection $\\tilde{f}_{i,1} = f_{i,1} + f^{\\mathrm{en}}_{i,s}$ into the first decoder layer's feed-forward activation; and (3) a Transform Matrix $W_T$, fit by least squares on 1,000 parallel pairs with closed-form solution, which maps non-English activations $F_i$ to English activations $F^{\\mathrm{en}}_i$ at inference. The first two parts carry the training-time benefit; the third carries the practical, monolingual inference-time simulation.","core_discovery":"On its own terms, the paper's central claim is that cross-lingual alignment can be engineered at the activation level: a model jointly fine-tuned on multilingual data benefits when the feed-forward activation of an English auxiliary input is injected into the first feed-forward layer of the non-English input at the response start token. A trainable linear Decision Maker with Gumbel-Softmax selects which English layer's activation is beneficial, making the injection adaptive. Because the training objective stays the vanilla negative log-likelihood, the model and the Decision Maker learn together to use both linguistic resources. At inference, a closed-form least-squares Transform Matrix, fit on 1,000 parallel pairs, transforms non-English activations into their English counterparts so the same fusion can be simulated on monolingual input. The paper reports that this mechanism outperforms vanilla SFT and gives a strong latent-level alternative to data-level augmentation methods, with the caveat that the gain disappears when the auxiliary input is also non-English or when English dominates the training data.","pith_inferences":["Editorial inference: if the linear Transform Matrix is the bottleneck, per-layer or per-language transforms (or a small learned network) should reduce the inference gap, and this is directly testable with the paper's own MSE metric.","Editorial inference: the paper's best combinations with +MT hint that latent-level fusion and data-level augmentation are complementary; a combined pipeline may push multilingual performance further than either alone.","Editorial inference: the middle-layer concentration suggests that a cheaper intervention—transplanting or fusing only a small band of layers instead of the whole stack—could capture most of the benefit at lower compute."],"forward_implications":["Because CC-Tuning adds no training objective and only a tiny linear layer, it can be layered onto existing multilingual SFT pipelines without changing the loss or data schedule.","The Transform Matrix converges with roughly 1,000 parallel pairs, so the method remains practical in low-resource settings where full parallel corpora are unavailable.","Under the +EN setting the benefit disappears, implying the mechanism is specifically a cross-lingual bridge rather than a general activation-mixing trick.","The Decision Maker concentrates on middle layers, suggesting that cross-lingual transfer during fine-tuning is largely a middle-layer phenomenon for feed-forward activations."],"supporting_citations":[{"why":"Supplies the empirical motivation that English feed-forward activations can enhance non-English performance and the cross-lingual transplantation operation.","marker":"[Ye et al., 2024b]"},{"why":"Grounds the choice of feed-forward activations by showing that FFNs store factual knowledge.","marker":"[Dai et al., 2022]"},{"why":"Provides Gumbel-Softmax, used by the Decision Maker for differentiable layer selection.","marker":"[Jang et al., 2016]"},{"why":"Supplies the Aya multilingual instruction dataset used as the training corpus.","marker":"[Singh et al., 2024]"},{"why":"Defines the vanilla supervised fine-tuning objective that CC-Tuning builds on and compares against.","marker":"[Ouyang et al., 2022]"},{"why":"Defines the SDRRL self-distillation baseline that CC-Tuning is compared with.","marker":"[Zhang et al., 2024]"},{"why":"Documents the curse of multilinguality that motivates the method.","marker":"[Conneau et al., 2020]"},{"why":"Provides LLaMA-3.1-8B, one of the two base models used in all experiments.","marker":"[Dubey et al., 2024]"},{"why":"Provides Qwen2.5-7B, the other base model used in all experiments.","marker":"[Yang et al., 2024]"}],"fun_headline_variants":["CC-Tuning fuses English activations to improve multilingual fine-tuning","Activation-level cross-lingual fusion outperforms data augmentation","Adaptive fusion of English hidden states lifts multilingual SFT","Latent cross-lingual connection improves non-English LLM tuning","Trainable Decision Maker fuses English activations for better multilingual tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that one linear map, learned from 1,000 parallel pairs, can reliably turn a non-English model's internal feed-forward activations into the English activations that would have helped during training, across all layers and languages.","fun_headline_variants_meta":{"raw":{"variants":["CC-Tuning fuses English activations to improve multilingual fine-tuning","Activation-level cross-lingual fusion outperforms data augmentation","Adaptive fusion of English hidden states lifts multilingual SFT","Latent cross-lingual connection improves non-English LLM tuning","Trainable Decision Maker fuses English activations for better multilingual tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3457,"prompt_tokens":961,"completion_tokens":2496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2408}},"tokens_in":577,"tokens_out":2496,"duration_ms":18459,"temperature":1.0,"reasoning_tokens":2408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:55:56.072993+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test is to evaluate the Transform Matrix's per-language residual $\\|f_{i,l}W_T - f^{\\mathrm{en}}_{i,l}\\|_2$ for languages not among the 1,000 parallel pairs used to fit it. If typologically distant or low-resource languages show residuals one order of magnitude above the average, the single linear map assumption fails and the inference-time simulation is not genuinely cross-lingual. A second check is to compare CC-Tuning's result using true English parallel activations against its result using $W_T$ language-by-language: the paper reports small average $|\\Delta|$, but the claim would be falsified if the gap grows systematically with language distance.","supporting_citations":[],"review_version":1}