{"id":"f6efe10f-68c6-4fac-9ce9-a245248b07c0","arxiv_id":"2501.07523","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"KV-Fusion encodes each retrieved passage in parallel with identical local positions, fuses the key-value caches, and fine-tunes a decoder so answers do not depend on passage order.","lead":"KV-Fusion is a technique that makes retrieval-augmented question answering insensitive to the order of the retrieved passages, by encoding each passage separately with identical position information and fusing their key-value caches. A generalist reader might care because position-invariant retrieval could make answer systems more stable and remove the need for rerankers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported accuracy gains are confounded: KV models are fine-tuned on gold answers plus evidence, while baselines are zero-shot; the position-invariance itself is architecturally guaranteed, so the key unverified claim is whether fusion, not SFT, drives the Table 1 improvements.","rationale":"The reader's weakest assumption identifies the same concern I find most load-bearing: the absence of a fine-tuned sequential baseline confounds every accuracy and robustness comparison. I examined whether a more fundamental flaw exists in the invariance claim itself. The mechanism's permutation invariance is real: because each context is encoded in isolation with identical positional embeddings, and the trainable decoder attends over the concatenated cache without any order-dependent masking, the attention outputs are mathematically symmetric under passage permutation. Therefore the 'consistent outputs' headline is not at risk. What is at risk is the paper's value proposition—that this architecture yields better RAG accuracy and robustness than existing approaches. The reported margins in Table 1 and Figure 4 compare a heavily fine-tuned model (with gold answers and evidence in its training targets) against zero-shot baselines. A reviewer cannot tell whether the 21.4-point NQ shuffled gain comes from position-invariant fusion or simply from task-format SFT. The concrete test I propose would settle this by adding the missing control; if the sequential SFT baseline matches KV-Fusion, the empirical contribution largely evaporates, leaving only the (already known) idea of parallel context encoding. This supports the reader's CONDITIONAL verdict, so no adjustment is needed.","tokens_in":14083,"tokens_out":7286,"duration_ms":64972,"concrete_test":"Fine-tune Llama3-8B (and Llama3.1-8B) on the same training instances used for the KV models, but with the 20 passages concatenated into a single standard context (natural positions, same order as the training set) and using the identical target format 'Answer: {ANSWER} [RESULT] Evidence: {EVIDENCE} [END]' and the same hyperparameters (2e-5, 2 epochs, etc.). Evaluate on the same shuffled dev sets (the 500-instance sets from Section 4). If this sequential SFT baseline's shuffled EM accuracy is within, say, 2 points of KV-Llama3/KV-Llama3.1 on all three datasets, then the fusion mechanism accounts for little of the reported improvement and the headline claim of superior robustness is not supported. If the KV models beat it by the reported margins, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim has two components: (1) KV-Fusion produces order-invariant outputs, and (2) it improves accuracy and robustness in RAG. Component (1) is structurally guaranteed: each passage is encoded independently by Dp with identical positional embeddings (positions 1..n), so the reshaped KV cache is a multiset of (key, value) pairs; attention in Dt is permutation-invariant over keys, so the output distribution is exactly unchanged when passages are shuffled. Table 2's ~99% TLM is consistent with this (the small deviations are likely floating-point ordering effects).\n\nComponent (2) is the load-bearing empirical claim, and it is not established. In Table 1 and Figure 4, KV-Llama3 is fine-tuned on each dataset with the gold answer and evidence (supervised with answer tokens and GPT-4o-extracted evidence), while Llama3, REPLUG, and PAM-QA are used zero-shot or with their default settings. The target format also includes 'Answer: {ANSWER}' and 'Evidence: {EVIDENCE}', so the model is directly trained to output the gold answers. Consequently, the reported gains of 21.4%, 6.4%, and 6.6% in shuffled settings could in principle be produced by an ordinary sequential fine-tuned reader; the comparison does not isolate the fusion mechanism. The paper's own claim to 'train language models to be context-order invariant' is also slightly off—the invariance is in the architecture, not the training—but this is a framing issue.\n\nThe missing control is a same-data, same-target-format fine-tuned baseline that concatenates the 20 passages in a standard context window with natural positional embeddings, and only differs by not using parallel cache fusion. Without it, the central empirical claim is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KV-Fusion, a two-decoder architecture for decoder-only RAG readers. A frozen prefill decoder encodes each retrieved passage independently with identical local positional embeddings and produces per-passage KV caches that are concatenated along the token dimension; a trainable decoder then generates answers conditioned on these fused caches plus the question, using next-token prediction. The design ensures that shuffling the order of passages does not change the attention geometry, thus yielding position-invariant outputs by construction. Experiments on NQ, TriviaQA, and POPQA report EM accuracy, token-level match, robustness to more passages, and comparisons to REPLUG and PAM QA, claiming accuracy gains of 21.4, 6.4, and 6.6 percentage points in shuffled settings.","tokens_in":14465,"tokens_out":6040,"duration_ms":55490,"significance":"The architectural insight that identical local positions plus cache fusion without cross-passage attention yields exact order invariance is clean and should hold. The TLM metric is a sensible way to measure token-level consistency. However, the quantitative claims that KV-Fusion improves accuracy are not yet separable from the effects of supervised fine-tuning and the evidence-augmented target format, so the contribution's empirical value is unproven. If a properly controlled comparison confirms the gains, this would be a practical contribution to position-invariant RAG.","major_comments":[{"comment":"The KV models are fine-tuned on each dataset with gold answers and GPT-4o-extracted evidence, while Llama3, REPLUG, and PAM QA are evaluated zero-shot or with default settings. The reported accuracy differences (21.4%, 6.4%, and 6.6% in shuffled settings) could therefore be due entirely to supervised fine-tuning rather than to the cache-fusion mechanism. Please include a control reader fine-tuned on the same training data and target format (answer plus evidence) with standard sequential cross-attention over all passages, and compare it against KV-Fusion trained on the same data. Additionally, ablate the evidence target by training KV-Fusion without the evidence in the target to isolate its effect.","section":"§4, Table 1 and Figure 4"},{"comment":"The near-perfect TLM reported in Table 2 is structurally guaranteed by the architecture: each passage is encoded at identical local positions and the reshaped cache is a permutation-invariant multiset of key-value pairs, so the output distribution is exactly unchanged under passage shuffling (modulo floating-point ordering effects). The conclusion's statement that KV-Fusion 'trains language models to be context-order invariant' overstates the role of training; the invariance is a property of the inference-time construction. Please revise the framing to make clear that the empirical TLM result confirms the architecture rather than a learned behavior.","section":"§2, §5, and §7"},{"comment":"The input preprocessing differs in a way that may confound the comparison. For KV-Fusion, each passage is tokenized to a maximum length of 192 tokens with left truncation (removing tokens from the beginning of long passages), as stated in Appendix A.5. For the baselines, the paper does not report whether the same per-passage truncation is applied or whether the entire concatenated context is truncated at the sequence level. If baselines receive longer or differently truncated contexts, the accuracy gap may reflect input processing choices rather than the fusion mechanism. Please specify the exact input construction for all systems and ensure that the only difference between conditions is the attention/fusion mechanism.","section":"Appendix A.5 and §4"},{"comment":"The claim that KV-Fusion 'eliminates the need for ranking in the RAG pipeline' is based on a comparison where KV models use the original top-40 passages while baselines use the top-20 reranked passages, and the KV models are fine-tuned. This comparison mixes two variables (passage count and fine-tuning) and does not isolate the fusion mechanism. A fair comparison would evaluate all systems on the same number of passages (e.g., top-20 reranked) and with the same fine-tuning status, or at least include an SFT baseline on concatenated contexts. Also, no variance or significance testing is reported for any of the differences in Tables 1 and 3, so the stability of the gains is unclear.","section":"§5, Table 3"}],"minor_comments":[{"comment":"The word 'irrelevent' should be 'irrelevant'.","section":"Abstract"},{"comment":"The word 'preprend' should be 'prepend'.","section":"§3.2"},{"comment":"The prompt template contains a typo: 'inlcude' should be 'include'.","section":"Appendix A.2.1"},{"comment":"The capitalization of 'LLama3' is inconsistent with the rest of the paper; use 'Llama3' consistently.","section":"Table 3 caption"},{"comment":"The relationship to parallel context encoding (Yen et al., 2024) and Fusion-in-Decoder (Izacard and Grave, 2020) should be explicitly discussed to position the novelty of KV-Fusion relative to these prior works.","section":"§6"},{"comment":"In the TLM definition, the notation p and s is reused without explicit definition; please clarify that p and s are the full generated responses in the POS1 and Shuffled scenarios.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern raised by the reader is accurate: the empirical core of the paper is confounded by supervised fine-tuning and by differences in input preprocessing. The position-invariance result is architecturally guaranteed, so the paper's headline empirical contribution depends on a controlled comparison that is currently missing. The paper appears to be a straightforward decoder-only adaptation of parallel context encoding, so the novelty statement should be sharpened with a discussion of prior work. I recommend major revision pending a fair SFT baseline and clarification of the input construction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. The order-invariance claim is structurally sound: encoding each passage independently with identical local positions and concatenating the key-value caches removes cross-passage position effects, so the near-100% token-level match is exactly what the architecture guarantees. The accuracy gains in Table 1, however, are not established by the experiments as written, because KV-Llama3 is fine-tuned on gold answers plus GPT-4o-extracted evidence while Llama3, REPLUG, and PAM-QA are used zero-shot or with default settings. That confound means the reported improvements over baselines could be produced by supervised fine-tuning alone, not by the fusion mechanism.\n\nWhat is genuinely new: the trained KV-Fusion models, the stability across up to 40 passages, and the reranker comparison. The paper also deserves credit for an honest limitations section that narrows the scope to single-hop QA. The method is simple and clearly described, and the position-invariance property is demonstrated convincingly at the token level.\n\nThe soft spots are real but fixable. First and most important, there is no same-data fine-tuned baseline that concatenates the 20 passages into a standard context window; without that control, the central empirical claim is unverified. Second, no code or data is released, and no error bars are reported. Third, the related work under-cites the core mechanism: Yen et al. 2024 is cited only for the Wikipedia corpus, but parallel context encoding is essentially the same idea, and the citation gap makes the novelty framing feel wider than it should. The conclusion also says the models are trained to be context-order invariant, but the invariance is in the architecture, not the training; that is a framing issue, not a technical one.\n\nWho is this for? RAG researchers who want an order-invariant reader without reranking. The paper is worth a serious referee, but the reviewers should push for a properly controlled baseline and for code/data release before the accuracy claims are taken at face value.","headline":"The order-invariance result is real but largely architectural; the accuracy gains over baselines are confounded by a missing same-data fine-tuned control.","tokens_in":15013,"tokens_out":1840,"would_cite":true,"duration_ms":20448,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KV-Fusion makes decoder-only LLMs order-invariant in RAG by encoding each retrieved passage with identical positional information and prefilling a trainable decoder with the fused key-value caches.","keywords":["key-value cache","position invariance","retrieval-augmented generation","lost in the middle","open-domain question answering","decoder-only language model","parallel context encoding","KV-Fusion"],"falsifier":"Fine-tune a standard Llama3-8B on the same training instances used for KV-Fusion (one gold passage, 19 negative passages, and the evidence string in the target) with ordinary sequential encoding, then compare shuffled-context exact match and token-level match; if the standard model also reaches near-perfect token-level match, the cache fusion is not the cause.","tokens_in":13874,"feed_emoji":"📌","tokens_out":8867,"duration_ms":74218,"temperature":0.7,"pith_summary":"Retrieval-augmented question answering fails when the one passage holding the answer sits in the middle of a long context. This paper tries to establish that this 'lost in the middle' failure can be trained away for decoder-only models by changing how retrieved passages are presented: each passage is encoded separately with the same positional slots, and a trainable decoder reads the fused key-value caches instead of a sequential token stream. On NQ, TriviaQA, and POPQA, the resulting models keep accuracy essentially constant as the gold passage moves, while base Llama3 models droop in the middle and drop sharply when contexts are shuffled; reported gains over baseline models in shuffled settings are 21.4%, 6.4%, and 6.6%. If the explanation is right, RAG pipelines could tolerate many irrelevant passages and skip reranking without losing accuracy.","feed_headline":"Shuffling retrieved passages no longer changes RAG answers","feed_subtitle":"KV-Fusion encodes every passage with identical positions, so decoder-only LLMs ignore context order in open-domain QA.","key_machinery":"The load-bearing object is the reshaped key-value cache. Per layer, the caches from all N passages are concatenated along the token axis, giving one cache block per layer that is injected into the trainable decoder's attention. This is what enforces position invariance: passage tokens always occupy positions 1 through n, and the instruction and answer tokens occupy positions n+1 onward, so reordering the passages does not alter any positional embedding seen by the decoder.","core_discovery":"The paper's central assertion is that decoder-only language models can be made context-order invariant for RAG by conditioning answer generation on fused key-value caches instead of on a linearly ordered prompt. A frozen prefill decoder reads each retrieved passage in parallel, so every passage is assigned the same local positions (1 through n); the layer-wise key and value caches are concatenated along the token axis and fed into a trainable decoder whose own query and answer tokens start at position n+1. Because a passage never changes its position when the passage list is shuffled, the model has no positional signal by which to behave differently, and the paper reports more than 99% token-level match between first-position and shuffled presentations, compared with low single-digit-to-teens match for standard Llama3 and Llama3.1.","pith_inferences":["Not tested in the paper: because the prefill decoder's caches depend only on the passages, the same fused cache could in principle serve many different questions about the same document set, which would make repeated RAG calls cheaper.","Not tested in the paper: the invariance is demonstrated for single-hop QA; whether cross-passage composition survives in multi-hop datasets like HotpotQA or MuSiQue is a direct consequence the authors flag as open.","If the mechanism is the cause rather than the fine-tuning data, the same cache-reshaping trick could be applied to continued pretraining, making order invariance a property of the base model rather than a per-task fine-tune."],"forward_implications":["Reranking becomes optional: KV-Fusion evaluated on the unranked top-40 passages matches or beats Llama3 and Llama3.1 evaluated on reranked top-20 passages on NQ, TriviaQA, and POPQA.","Accuracy stops degrading as context grows: KV-Fusion stays within a small margin between top-5 and top-40 passages, whereas PAM-QA loses roughly half its accuracy on average between top-20 and top-40.","Outputs become reproducible under shuffling: with greedy decoding, shuffled contexts produce token-identical answers in over 99% of cases, so downstream use of RAG outputs no longer depends on retrieval order.","The effect transfers across base models: the same training recipe gives position-invariant readers from both Llama3-8B and Llama3.1-8B."],"supporting_citations":[{"why":"Documents the 'lost in the middle' positional bias that the paper targets.","marker":"Liu et al., 2023"},{"why":"PAM-QA, the main position-agnostic baseline that KV-Fusion is compared against.","marker":"He et al., 2024"},{"why":"REPLUG, the retrieval-augmented black-box baseline it is compared against.","marker":"Shi et al., 2024b"},{"why":"Supplies the DPR retriever and the filtered NQ and TriviaQA datasets used for training and evaluation.","marker":"Karpukhin et al., 2020"},{"why":"Source of the Natural Questions dataset used as one evaluation benchmark.","marker":"Kwiatkowski et al., 2019"},{"why":"Source of the TriviaQA dataset used as one evaluation benchmark.","marker":"Joshi et al., 2017"},{"why":"Source of the POPQA dataset used as one evaluation benchmark.","marker":"Mallen et al., 2023"},{"why":"Provides the Llama3-8B and Llama3.1-8B models that are initialized and fine-tuned.","marker":"Dubey et al., 2024"},{"why":"Parallel context encoding work that motivates the per-passage independent encoding design.","marker":"Yen et al., 2024"},{"why":"MonoT5 reranker used in the comparison showing reranking becomes unnecessary.","marker":"Nogueira et al., 2020"}],"fun_headline_variants":["Position-proof RAG: fused KV caches ignore context order","Order-blind answers: parallel KV fusion fixes lost-in-the-middle","KV fusion makes RAG immune to passage shuffling","Parallel cache fusion gives RAG position invariance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that fine-tuning on gold answers plus extracted evidence is not itself what produces the stable, order-insensitive behavior, because the KV-Fusion models receive that fine-tuning while Llama3, REPLUG, and PAM-QA do not.","fun_headline_variants_meta":{"raw":{"variants":["Position-proof RAG: fused KV caches ignore context order","Order-blind answers: parallel KV fusion fixes lost-in-the-middle","KV fusion makes RAG immune to passage shuffling","Parallel cache fusion gives RAG position invariance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1248,"prompt_tokens":807,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":423,"tokens_out":441,"duration_ms":4463,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:40:08.756805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a standard Llama3-8B on the same training instances used for KV-Fusion (one gold passage, 19 negative passages, and the evidence string in the target) with ordinary sequential encoding, then compare shuffled-context exact match and token-level match; if the standard model also reaches near-perfect token-level match, the cache fusion is not the cause.","supporting_citations":[],"review_version":1}