{"id":"24fabd55-16a7-4673-b34f-3236ea6ba677","arxiv_id":"2501.06892","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FLARE fuses source and target language representations inside LoRA adapter bottlenecks, improving cross-lingual question answering by up to 4.9% exact match over standard LoRA.","lead":"This paper introduces FLARE, a method that combines English and non-English language representations inside small adapter modules during multilingual model fine-tuning. It reports gains over standard fine-tuning on several cross-lingual tasks, especially question answering, while adding almost no extra parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's LoRA comparison is confounded: FLARE sees the English source sentence during training while LoRA does not; the source-data control (input-level fusion) is not significantly beaten on TyDiQA.","rationale":"The reader's stated weakest assumption (positional alignment after machine translation) is a genuine mechanism concern, but it is not the most load-bearing for the central claim: even imperfect alignment could still yield empirical gains through regularization or coarse cross-lingual signal, and the empirical comparison would still stand. The confound between fusion and source-data availability directly threatens the attribution of the improvement to fusion itself. The paper deserves credit for releasing code, covering three tasks and four model families, and reporting permutation tests, but the missing control is inexpensive and would settle the interpretation. This is an addressable experimental gap rather than a reason to reject. I therefore keep the reader's CONDITIONAL verdict, though with a different emphasis; the reader's rationale did mention the source-data confound, so agreement is partial.","tokens_in":27498,"tokens_out":6638,"duration_ms":70822,"concrete_test":"Add a control baseline: standard LoRA fine-tuned on the same parallel corpus by alternating batches of English source and NLLB-translated target examples (same total steps, same LoRA r/alpha, same five seeds), with no token-wise fusion and no use of source hidden states. If this control reaches FLARE's TyDiQA and NusaX scores within one standard deviation, the headline gains over LoRA are explained by source-data exposure. If FLARE still beats this control significantly in a paired permutation test over languages (as in Table 9), the fusion mechanism is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FLARE's central claim is that fusing English and target representations inside LoRA bottlenecks is what drives the reported gains. The training procedure in Section 3.3 gives FLARE access to the English source sentence on every training step: the frozen base model processes xS, and its hidden states are fused into the target adapter at every layer. The 'standard LoRA fine-tuning' baseline sees only the target-language translation. The +4.9/+2.2 EM gains in the abstract are therefore not attributable to the fusion mechanism alone; they may reflect the additional source-language data and its representations. The input-level fusion baseline does use both languages, but it concatenates the two sequences and doubles the input length, which is a different computational and inductive setup. On the headline TyDiQA task, Table 9 shows FLARE versus input-level fusion is not significant (Llama p=0.378, Gemma p=0.505), and Table 9 also shows FLARE versus LoRA is non-significant in several other settings (e.g., XLM-R TyDiQA p=0.296, mT5 TyDiQA p=0.181). Thus the claim that adapter-level fusion, rather than source-data availability, causes the improvement is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLARE, a parameter-efficient method for cross-lingual transfer that fuses English (source) and target-language representations inside LoRA adapter bottlenecks. During fine-tuning, source representations are extracted from the frozen base model at each transformer block, down-projected, combined with target representations via lightweight functions such as addition, multiplication, ReLU, or cross-attention, and then up-projected and added to attention outputs. The method is evaluated on XNLI, TyDiQA, and NusaX using XLM-R Large, mT5-XL, Llama 3.1 8B, and Gemma 2 9B, and compared against zero-shot cross-lingual transfer, translate-test, translate-train with standard LoRA, X-Mixup, and input-level fusion. The headline results are improvements on TyDiQA exact match over standard LoRA, e.g., +4.9% for Llama 3.1 and +2.2% for Gemma 2, with no added parameters in the default linear-fusion configuration. The paper also introduces FLARE MT, which fuses latent translations from an MT encoder into the adapters.","tokens_in":27778,"tokens_out":4455,"duration_ms":47328,"significance":"If the reported gains are reliable, FLARE is a potentially valuable contribution to parameter-efficient cross-lingual transfer: it keeps the frozen backbone intact, adds no parameters beyond standard LoRA in its default variant, and works across encoder-only, encoder-decoder, and decoder-only architectures. The experimental campaign is unusually broad for this area, with four model families, three tasks, five seeds, permutation tests, and a honest train-only ablation (Appendix E) that shows source-language representations must be present at inference for FLARE to work. However, the central causal claim—that fusing source and target representations inside the adapter rather than simply having access to source-language data drives the gains—is not yet established because the principal comparison is confounded and several headline differences are not statistically significant. The significance of the paper therefore depends on whether the authors can close this gap with additional controls and more careful claims.","major_comments":[{"comment":"The comparison between FLARE and 'standard LoRA fine-tuning' is confounded by source-language data availability. FLARE's forward pass processes the English source sentence xS at every training step and fuses its hidden states into the target adapter at every layer, whereas the LoRA baseline in the translate-train setting is trained only on translated target-language data. The gains in Table 1 (e.g., +4.9 exact match for Llama 3.1 on TyDiQA) may therefore reflect the extra information provided by seeing the English source, not the fusion mechanism itself. The input-level fusion baseline does use source and target data, but it concatenates the sequences and doubles the input length, which is a different computational and inductive setup. To isolate the fusion mechanism, please add a control in which the model also has access to the English source at training time but does not fuse layer-wise in the adapters—for example, pass the source through a frozen encoder and inject its pooled or projected representations only at the task head, while otherwise training a standard LoRA adapter on the target data. If such a control matches FLARE's performance, the central claim in the abstract would need to be reframed.","section":"Section 3.3 and Section 4.1"},{"comment":"The statement that 'FLARE consistently surpasses all baselines across various tasks' is not supported by the reported permutation tests. Table 9 shows many non-significant differences: FLARE vs. LoRA is non-significant on XLM-R TyDiQA (p=0.296), mT5-XL TyDiQA (p=0.181), Llama XNLI (p=0.321), and Gemma XNLI (p=0.221); FLARE vs. input-level fusion is non-significant on Llama TyDiQA (p=0.378), Gemma TyDiQA (p=0.505), and XLM-R TyDiQA (p=0.531). The only consistently significant gains over LoRA are on TyDiQA for the decoder-only models. Please report the significance status for every headline average in Table 1, or qualify the 'consistently surpasses' claim to reflect the actual support. The abstract's unqualified framing would otherwise overstate the evidence.","section":"Section 5 and Table 9"},{"comment":"The token-wise fusion functions are defined positionally: S and T are down-projected source and target representations at the same token position, and the method's effectiveness is interpreted as evidence of positional alignment (Figures 5 and 7, and the discussion of ReLU as a way to 'restrict the propagation of misaligned information'). The paper does not, however, measure alignment quality or test how sensitive the result is to misalignment. Please quantify this assumption—for example, by comparing FLARE with source tokens shuffled against their aligned order, or by measuring token-level correspondence with MT alignment tools. This would also clarify why the gold-translation results in Table 4 show FLARE substantially underperforming input-level fusion on NusaX despite having perfect English input.","section":"Section 3.2 and Figure 7"}],"minor_comments":[{"comment":"This subsection refers to 'Table 2' for latent MT fusion results, but Table 2 reports the fusion-function ablation; the FLARE MT results are given in Table 1 and the significance tests in Table 9. Please correct the cross-reference.","section":"Section 5, 'On Latent MT Fusion'"},{"comment":"The abstract says FLARE combines representations 'using lightweight linear transformations', but Section 3.2 also evaluates a cross-attention fusion function that introduces additional parameters. Please clarify that the parameter-free claim applies to the default linear/reLU variants, not to all evaluated fusion functions.","section":"Abstract and Section 3.1"},{"comment":"The table reports p-values from Pitman permutation tests, but the caption does not state whether the tests are two-sided or one-sided, nor how the 90/95/99% thresholds correspond to *, **, ***. Please add these details.","section":"Table 9 title"},{"comment":"The train-only ablation is valuable and honest, but the sentence 'the performance of the train-only FLARE variant decreased by 30%' should state whether 30% is a relative or absolute decrease and on which average across languages this is computed.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically solid and the experimental effort is impressive, but the central causal claim needs either a dedicated control experiment or a substantial softening. I would not require an entirely new experimental campaign; a targeted control that gives a standard LoRA baseline access to the same source-language information, plus a revised significance-aware framing, would be sufficient. I would also encourage the authors to reconsider the phrase 'consistently surpasses all baselines' in the light of Table 9."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nick, here's my take on Borchert et al. (arXiv:2501.06892). The short version: this is a solid empirical paper with a genuinely new mechanism, but the headline comparison overstates what is proven. The idea is to fuse English and target-language representations inside LoRA bottlenecks, token-wise, using parameter-free linear functions. That is not in prior work: X-Mixup fuses in one layer with cross-attention and extra parameters, and input-level fusion just concatenates the two sequences. FLARE processes the two languages separately, down-projects both, fuses in the bottleneck with add/mul/ReLU, and up-projects. No extra parameters. That's clean and useful.\n\nWhat the paper does well: broad and careful evaluation — four model families (encoder-only, encoder-decoder, two decoder-only LLMs), three tasks, five seeds, permutation tests on per-language scores. The FLARE MT variant (fusing MT encoder latents directly) is a nice efficiency idea. There is an honest train-only ablation showing that source representations matter at inference, and a gold-vs-MT comparison showing FLARE is more robust to translation quality than input-level fusion. Code is public.\n\nThe soft spot is the comparison that makes the headline: 'vs standard LoRA fine-tuning.' The LoRA baseline only sees the target-language sentence; FLARE sees the English source on every training step and at inference. So part of the +4.9/+2.2 EM gain is simply having extra source-language data. The right control is input-level fusion, which also sees both languages. On TyDiQA, FLARE does not beat input-level fusion significantly for Llama (p=0.378) or Gemma (p=0.505). It does beat it on some tasks/models, and it is reliably better than X-Mixup. But the specific claim that adapter-level fusion, rather than source-data availability, drives the gain is only partially supported.\n\nOther minor issues: fusion-function and rank ablations are reported without error bars, and the token-alignment assumption (Section 3.2) is not directly measured — the ReLU variant is a reasonable hedge but the paper doesn't quantify how often source and target are misaligned. These are minor compared to the confound.\n\nBottom line: this is a paper for people working on cross-lingual PEFT. It deserves peer review, and a serious referee should push for a source-data control (e.g., LoRA trained on source+target text without fusion) and per-language significance tables. The mechanism is worth publishing; the title claim just needs to be calibrated. I'd bring it to reading group and would cite it if I worked in this area.","headline":"Solid new fusion mechanism with a confounded headline comparison; worth review, but the LoRA gains partly reflect extra source data.","tokens_in":28273,"tokens_out":3095,"would_cite":true,"duration_ms":30133,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing English and target-language representations inside LoRA adapter bottlenecks improves cross-lingual downstream performance, with the largest gain on question answering.","keywords":["cross-lingual transfer","parameter-efficient fine-tuning","LoRA","adapter fusion","representation fusion","low-resource languages","multilingual question answering"],"falsifier":"Permute the source-language token order before fusing with the target representation inside the adapter bottleneck and measure downstream accuracy: if the shuffled-source result stays close to aligned FLARE, then token-level alignment is not carrying the transfer and the method's stated mechanism is not responsible for its gains.","tokens_in":27321,"feed_emoji":"🌐","tokens_out":4773,"duration_ms":50339,"temperature":0.7,"pith_summary":"This paper tries to establish that cross-lingual transfer can be done by merging English and target-language representations inside the low-rank bottleneck of LoRA adapters, rather than by mixing text at the input level or adding new model parameters. It argues that this representation-level fusion lets task knowledge flow from a high-resource language into a target language during fine-tuning, and that the mechanism works across encoder-only, encoder-decoder, and decoder-only multilingual models. The payoff is parameter-efficient transfer: FLARE adds no parameters beyond standard LoRA, avoids the doubled sequence lengths of input-level fusion, and still beats LoRA fine-tuning, X-Mixup, and input-level fusion on natural language inference, sentiment analysis, and question answering. A sympathetic reader would take the central claim to be that the adapter bottleneck is not just a compression device but a place where two languages' representation spaces can be productively combined.","feed_headline":"Fusing languages inside LoRA adapters lifts multilingual QA by 4.9%","feed_subtitle":"Blending English representations into target languages at each layer beats LoRA fine-tuning, with no extra parameters.","key_machinery":"The central object is the fusion adapter: a LoRA adapter whose down-projection maps both source and target hidden states into a shared low-dimensional bottleneck, where they are combined token-wise by a fusion function, after which the shared up-projection returns the result to full model dimension and adds it to the frozen attention output. Formally, the fused representation is $h=\\varphi(v^S_{i+1}W_{\\text{down}},\\; v^T_i W_{\\text{down}})$, with $v^S$ taken from the following transformer block to capture task-specific source information, and the final output is $v^T_{i+1}=hW_{\\text{up}}+v_0$. The fusion functions examined are element-wise addition, element-wise multiplication, a cross-attention variant, and their ReLU-gated versions, with addition plus ReLU giving the best average results. This mechanism does the work of transferring knowledge because it keeps the two languages in separate computation streams until the moment of fusion, then forces their aligned token representations to interact in the compressed rank space that the task head actually reads from.","core_discovery":"The paper introduces FLARE, which runs the source-language input through the frozen, adapter-free base model and the target-language input through the same model with LoRA adapters, extracts layer-wise hidden states from each, and fuses them inside the adapter bottleneck through a lightweight function such as element-wise addition or multiplication, optionally followed by ReLU. The fused low-rank representation is then up-projected and added to the frozen attention outputs. This token-wise fusion is applied in every transformer block, so the model continuously mixes source and target information during task adaptation instead of only in one layer or only at the input. Across XNLI, NusaX, and TyDiQA with XLM-R, mT5, Llama 3.1, and Gemma 2, FLARE outperforms standard LoRA fine-tuning on all three tasks, with the largest gains on question answering: +4.9 exact-match points for Llama 3.1 and +2.2 for Gemma 2 on TyDiQA. A variant called FLARE MT replaces the source-language forward pass with a latent translation from an MT encoder, cutting compute further at some performance cost for decoder-only models.","pith_inferences":["If token-wise alignment is genuinely the mechanism, then a direct test is to shuffle or shift the source token order before fusion; a large performance drop would confirm that positional matching carries the effect, while a small drop would suggest the method mainly injects extra features.","FLARE is explicitly agnostic to the source language, but the paper only uses English; the same adapter fusion could plausibly transfer from another high-resource language, such as Indonesian for NusaX languages, to related low-resource languages, which the paper itself lists as untested.","The fact that small bottlenecks (rank 8) work best on XLM-R hints that the cross-lingual signal is low-dimensional, which would imply that even cheaper adapters or other PEFT formulations could host the same fusion step.","Since FLARE tolerates lower-quality machine translations, a practical extension is to use it as a training recipe that lets smaller MT models suffice for building task data in low-resource languages, thereby lowering the compute budget for rapid-language adaptation."],"forward_implications":["FLARE's biggest wins are on generative question answering with decoder-only models, raising exact match by 4.9 points on Llama 3.1 and 2.2 points on Gemma 2 over standard LoRA fine-tuning.","The method is architecture-agnostic: it improves translate-train performance on XNLI and NusaX for encoder-only XLM-R, encoder-decoder mT5, and decoder-only Llama and Gemma, so the fusion idea transfers across model families.","Replacing the source-language forward pass with an MT encoder's latent translation gives a cheaper variant that still beats LoRA on encoder-style models, meaning cross-lingual transfer can be done without decoding a full source sentence.","FLARE remains competitive when the machine translation quality is lower (using NLLB 600M instead of 3.3B), suggesting that the fusion bottleneck tolerates noisier parallel data than the comparison methods.","Fusing only during training collapses NusaX performance by 30%, so the source-language representations are needed at inference time; the transfer is instance-dependent, not a static pattern learned from English data alone."],"supporting_citations":[{"why":"Supplies the LoRA adapter architecture that FLARE builds on, including the down-projection bottleneck and up-projection used for fusion.","marker":"(Hu et al., 2022)"},{"why":"Provides the prior work on routing functions inside low-rank bottlenecks and informs the ReLU-gated fusion functions FLARE evaluates.","marker":"(Qu et al., 2025)"},{"why":"Introduces X-Mixup, the main representation-mixing baseline FLARE must outperform, which uses cross-attention and consistency losses in a single layer.","marker":"(Yang et al., 2022)"},{"why":"Represents the input-level fusion baseline that concatenates source and target texts, which FLARE is designed to beat on both quality and computational cost.","marker":"(Cueva et al., 2024)"},{"why":"Provides the machine translation system that generates the parallel training data and the latent translations used in the FLARE MT variant.","marker":"(NLLB Team et al., 2022)"},{"why":"Supplies the TyDiQA dataset where FLARE shows its largest gains, grounding the question-answering results.","marker":"(Clark et al., 2020)"},{"why":"Supplies NusaX, the low-resource sentiment dataset used to test FLARE's behavior when training data is scarce.","marker":"(Winata et al., 2023)"}],"fun_headline_variants":["Fusing languages in LoRA adapters lifts cross-lingual QA by 4.9%","Cross-lingual QA up 4.9% by fusing representations in LoRA adapters","No new parameters: FLARE blends languages in adapters for better QA","Adapter fusion boosts cross-lingual QA, keeping parameters flat","FLARE: merge languages in adapters for parameter-efficient multilingual gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"For the fusion to transfer meaningful information, the down-projected source and target representations must be aligned token by token after machine translation, so that adding or multiplying them at each position pairs up corresponding meanings.","fun_headline_variants_meta":{"raw":{"variants":["Fusing languages in LoRA adapters lifts cross-lingual QA by 4.9%","Cross-lingual QA up 4.9% by fusing representations in LoRA adapters","No new parameters: FLARE blends languages in adapters for better QA","Adapter fusion boosts cross-lingual QA, keeping parameters flat","FLARE: merge languages in adapters for parameter-efficient multilingual gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000999,"raw_usage":{"total_tokens":4252,"prompt_tokens":993,"completion_tokens":3259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3154}},"tokens_in":609,"tokens_out":3259,"duration_ms":23209,"temperature":1.0,"reasoning_tokens":3154,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:41.456042+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the source-language token order before fusing with the target representation inside the adapter bottleneck and measure downstream accuracy: if the shuffled-source result stays close to aligned FLARE, then token-level alignment is not carrying the transfer and the method's stated mechanism is not responsible for its gains.","supporting_citations":[],"review_version":1}