{"id":"613cf132-f902-41dd-bcab-9308d01fc5e4","arxiv_id":"2607.28979","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MoT translates KV caches between LLMs using token-level translator mixtures and a context-correction loss; QA results are strong for same-family pairs but do not demonstrate true cross-architecture reuse.","lead":"The paper proposes MoT, a system that translates the internal KV cache of one language model into the cache space of another, so the second model can reuse context without re-reading it. It works in same-family model pairs and reduces memory in multi-agent settings, but the \"heterogeneous\" headline is not tested across truly different architectures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MoT's 'heterogeneous LLM' claim is not actually stress-tested: every QA experiment is within one model family, and the depth-ratio channel mapping the pipeline depends on is admitted (App. C.3, H) to weaken across genuinely different architectures/tokenizers.","rationale":"The reader's weakest assumption is exactly this: depth-ratio mapping is never stress-tested across genuinely heterogeneous architectures or tokenizers. I agree, and I consider it the most load-bearing concern because the paper's title, abstract, and conclusion promise cross-architecture KV-cache reuse, while every benchmark that supports the empirical claim is within a single model family (Qwen2.5, GPT-2, or OPT). The 'heterogeneous' settings are same-family size/scale differences, not the attention-layout or tokenizer differences that motivate the paper. The paper itself flags the limitation in App. C.3 and H, but does not provide cross-family downstream results to bound it. The empirical within-family results are nevertheless real support: MoT performs well on several benchmarks, ablations isolate the effects of the mixture and correction loss, and the theoretical claims are mostly harmless superset or Lipschitz arguments. Those facts justify a conditional acceptance rather than rejection: the architecture is plausible, but the headline generalization is untested. No error bars or released code/data further weaken the precision of the empirical claim, but the decisive missing evidence is a genuinely cross-family experiment. One concrete cross-family run, ideally with and without tokenizer mismatch, would settle whether the depth-ratio channel mapping and MoT's token-level translation survive real architectural heterogeneity. Since the reader already set CONDITIONAL and my concern is the same one, the verdict should remain unchanged.","tokens_in":34828,"tokens_out":7434,"duration_ms":61677,"concrete_test":"Train and evaluate MoT on a genuinely cross-family pair with different tokenizers—e.g., Qwen2.5-7B as source and LLaMA-3.2-3B (or Pythia-2.8B) as target—using the same E.1 recipe and the same BoolQ/SQuAD evaluation. If MoT's accuracy/F1 gap to Native grows from roughly 2% (Table 1) to more than ~20%, or if the depth-ratio-chosen window fails to produce a usable channel set, the central heterogeneity claim is not supported. Repeat with a matched tokenizer (e.g., Qwen→Pythia with a shared tokenizer if available) to isolate tokenizer mismatch from depth-ratio mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MoT translates KV caches across heterogeneous LLMs. The load-bearing condition is that the channel set C, built by Depth-Ratio Mapping (App. C.3), connects semantically corresponding layers between source and target. All downstream QA experiments in §4.2 and App. E.2 are within one family: Qwen2.5-7B→0.5B, gpt2-xl→large, opt-6.7b→125m. These vary depth, width, and KV heads, but not attention layout, tokenizer, or training-family mismatch. The only cross-architecture evidence is the layer-similarity analysis in Fig. 15, which is not a downstream evaluation, and Fig. 15(e) already shows the depth-ratio pattern can break under instruction/data mismatch. App. H explicitly says depth-ratio alignment weakens when models differ in instruction tuning, training data, or representation organization, and that cross-tokenization translation is not addressed. Since the channel set is fixed before translation and the whole pipeline (translation, replay, correction loss) operates relative to that C, a failure of depth-ratio correspondence is not a minor parameter; it invalidates the channel selection and with it the claimed cross-architecture generalization. Thus the abstract's 'across heterogeneous LLMs' is supported only for same-family size heterogeneity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Mixture-of-Translators (MoT), a method for translating a source LLM's KV cache into the cache space of a target LLM. The method uses token-level gated combinations of backbone cross-attention translators and a Context Correction Loss that replays the translated cache and aligns it with the native target trajectory. The authors identify two competing failure modes—propagated translation shift from early injection and correction-deficit error from late injection—and provide analytic bounds and a shift decomposition (Props. 2.3–2.5). They evaluate on BoolQ, PubMedQA, MMLU-Redux, SQuAD, NewsQA, plus multi-agent and long-context CAG case studies, reporting that MoT roughly matches native performance in same-family settings (e.g., Qwen2.5-7B→0.5B: 51.0% accuracy, 0.43 F1) and retains 96.3% of native F1 in long-context CAG.","tokens_in":35219,"tokens_out":9911,"duration_ms":80309,"significance":"If the results hold, MoT is a useful step toward KV-cache reuse in multi-model systems: it addresses a real bottleneck, and the competing-error analysis is a reasonable design lens. The paper compares against multiple baselines, provides detailed ablations and case studies, and is unusually candid in its limitations. However, the headline 'heterogeneous' claim is currently supported only for same-family depth/width variation, not cross-family or cross-tokenizer translation, and the headline numbers are single-run. The idea is promising, but the empirical scope and statistical reliability need to be strengthened before the abstract's claims are warranted.","major_comments":[{"comment":"The central claim 'across heterogeneous LLMs' is not tested in the downstream evaluations. Table 1, Tables 5–6, and Fig. 28 use only same-family pairs: Qwen2.5-7B→0.5B, gpt2-xl→large, and opt-6.7b→125m. There is no Qwen→GPT, GPT→OPT, or any cross-tokenizer translation. The channel set C on which the whole pipeline depends is fixed by Depth-Ratio Mapping (App. C.3), and the paper itself says this mapping weakens under instruction-tuning/data/tokenizer mismatch and that cross-tokenization is not addressed (App. H; Fig. 15(e)). Thus the abstract's 'heterogeneous LLMs' claim is supported only for size heterogeneity within a family. Please either add cross-family/cross-tokenizer experiments or narrow the claim.","section":"Abstract; §4.2; App. C.3; App. H"},{"comment":"All reported numbers appear to be single runs. Headline differences are small: Table 1 has MoT 51.0% vs Native 52.0% accuracy, and Fig. 31 has MoT 0.2047 vs Native 0.2125 F1 (96.3% retention). With 500 training steps and batch size 4 (App. E.1), these gaps are within normal run-to-run variability. Please report multiple seeds or confidence intervals for at least the main QA tables and the 96.3% claim; otherwise 'preserves performance' is not statistically supported.","section":"Tables 1, 5, 6; Fig. 31"},{"comment":"The OPT capacity-scaling analysis excludes opt-125m as a source because it 'behaves as an outlier.' Excluding an inconvenient data point without explanation undermines the claim that MoT remains stable as source capacity varies. Please report the excluded values and justify the exclusion on a stated criterion (e.g., convergence failure) or include the point in the analysis.","section":"App. E.5, Fig. 28"},{"comment":"Proposition C.1 assumes the stacked final-layer key/value projection has full column rank. This fails for GQA models such as Qwen2.5-7B, where the number of KV heads is smaller than the number of query heads: the stacked W^KV_L matrix has more columns than rows and cannot be full column rank. Therefore the proof that reducing the final-layer KV matching error reduces an upper bound on ||s_L|| does not apply to the main experimental model. The loss may still be a useful heuristic, but the theoretical justification should be restricted or the assumption verified.","section":"App. C.1, Prop. C.1"}],"minor_comments":[{"comment":"Proposition 3.2 states that the optimal MoT cannot be worse than the optimal single translator because the MoT class contains the single translator as a special case. This is true by construction and should be described as a containment property, not a substantive reduction theorem. The empirical evidence in Table 2 is what actually supports the claim.","section":"§3.1, Prop. 3.2"},{"comment":"The phrase 'scale-invariant memory' is too strong: MoT(Free) offloads non-hub caches to the hub, so total stored cache across the system may still grow; only the active peak memory is bounded. Please clarify the metric being reported.","section":"§5, Fig. 13"},{"comment":"C2C-Project and LSC are 'implemented by us' with no code or detailed hyperparameters beyond the shared setup. Please provide exact configurations or release code so the baselines can be verified.","section":"§4.1, App. E.1"},{"comment":"The term 'heterogeneous' is used both for within-family size differences (Qwen2.5-7B vs 0.5B) and for cross-architecture differences. Define the term at first use and keep the claim consistent with the experimental scope.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid engineering contribution and an honest limitations section, but the abstract overclaims cross-family generalization. I would be willing to reconsider if the authors add a genuinely cross-family/cross-tokenizer experiment or substantially narrow the claims; the current version should not be accepted as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The core evidence holds up for what it actually tests: within a model family, MoT translates a source KV cache into the target cache space and preserves QA performance close to native, beating the baselines. The token-level gated mixture plus the context-correction loss is a legitimate extension of prior cache-translation work (C2C, LSC, KVComm, Interlat), and the ablations are honestly done. The multi-agent and long-context case studies show real memory savings when you evict non-hub caches, though the translator overhead is not nothing.\n\nThe soft spot is exactly where the stress-test note lands. The abstract says 'across heterogeneous LLMs,' but every downstream QA experiment is within one family: Qwen2.5-7B→0.5B, gpt2-xl→large, opt-6.7b→125m. Those vary depth and width, not tokenizer, attention layout, or training-family mismatch. The only cross-family evidence is the layer-similarity analysis in Fig. 15, which is not a downstream evaluation and which, as the authors admit in App. C.3 and H, breaks under instruction/data mismatch. So the central claim is overbroad. The authors are upfront about this in the limitations section — credit where due — but the abstract and intro still sell it as general.\n\nSmaller issues: no error bars anywhere, and some of the reported gains (e.g., 51.0 vs 49.0) are within the range where single-run noise could matter. No code or data released. The theoretical propositions are mostly definitional or superset arguments; Proposition 3.2 is a 'mixture contains the single translator as a special case' argument, which is true but not deep. That is fine, but it shouldn't be sold as a main contribution.\n\nI'd send this to peer review. The task is real, the within-family results are useful, and the architecture is a reasonable step. But the authors need to either test across genuinely different model families and tokenizers, or re-scope the claims to 'within-family size heterogeneity' and say why that is still useful. With that change plus error bars, this could be a solid systems paper.","headline":"Decent within-family cache translation with an oversold heterogeneity claim; deserves a serious referee after a major re-scope.","tokens_in":35686,"tokens_out":2568,"would_cite":false,"duration_ms":23782,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that key-value caches of one LLM can be translated into another LLM's cache space with almost no quality loss, by attacking two opposing failure modes at once.","keywords":["KV cache translation","mixture-of-translators","context correction loss","heterogeneous LLM systems","cache reuse","multi-agent reasoning","cache-augmented generation","translation shift"],"falsifier":"Take a pair of models from genuinely different architecture families with different tokenizers, translate a context cache with the same MoT pipeline, and measure closed-set QA accuracy against direct prefill; if accuracy collapses to near random, the cross-model generality of the claim fails.","tokens_in":1226,"feed_emoji":"🔁","tokens_out":1635,"duration_ms":46839,"temperature":0.7,"pith_summary":"The paper tries to establish that the context key-value caches built by one large language model can be mapped into another model's internal cache space without destroying the information the target model needs to answer questions. It argues that naive translation fails for two competing reasons—errors injected too early propagate through the whole network, while errors injected too late leave too few upper layers to correct—and that a token-level mixture of translator modules plus a context-correction training signal can suppress both. If this is right, multi-agent reasoning and long-context systems would no longer need each model to prefill or store its own copy of the same shared context, which would cut peak memory and repeated computation. Across model pairs within the Qwen2.5, GPT-2, and OPT families, the approach is reported to preserve roughly 96% of direct-context quality and maintain stable QA accuracy for both homogeneous and heterogeneous settings.","feed_headline":"Cache translation across LLMs keeps 96% of answer quality","feed_subtitle":"A mixture-of-translators approach lets heterogeneous models share one context cache, cutting memory and duplicated work.","key_machinery":"The central object is the cache-translation channel set C, a collection of connected source–target layer pairs through which source KV states are injected into the target. MoT combines recurrent cross-attention backbone translators into a token-level Top-K gated mixture, so different tokens can use different translation mappings. The Context Correction Loss compares the replayed target KV cache (or hidden states) against the native one across the upper layers. Channel selection uses Depth-Ratio Mapping, which connects layers of similar relative depth, chosen by sliding a contiguous window and picking the lowest validation loss.","core_discovery":"MoT translates a source model's context KV cache into the target model's cache space using several translator modules with token-level gating, and trains them with a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory. The paper identifies two competing translation errors: a propagated translation shift that grows when translation happens too early, and a last-state shift that appears when translation happens too late and leaves too few upper layers for correction. It formalizes both as bounds under a residual-transformer recurrence and shows that an optimal translator mixture cannot increase the translation shift, while the context-correction","pith_inferences":["The depth-ratio channel mapping is the least-tested assumption: it is only stress-tested within a few model families, so a genuine cross-family or cross-tokenizer pair could still break the pipeline unless channel selection becomes learned or attention-based.","The context-correction loss mainly reduces the last-state shift by shrinking the translation shift and its orthogonal component, while the correction-deficit coefficient barely changes; explicitly training the target's own correction ability would be a natural next step.","Cross-tokenizer translation is unexplored; if token-position correspondence could be established, MoT would apply to the pairs where cache reuse is most valuable, since tokenizer mismatch is the main obstacle to truly general heterogeneous reuse.","The practical memory win depends on translation overhead staying smaller than prefill cost; the paper's fixed translator overhead suggests the break-even point comes with larger source models, and its activation-based variant already offers a leaner alternative."],"forward_implications":["A shared context prefix could be stored once in one model's cache format and translated on demand, eliminating duplicate prefills across heterogeneous models.","Multi-agent reasoning can offload completed agents' caches to a hub and free local memory, keeping peak memory nearly constant as the number of agents grows.","Long-context cache-augmented generation could reuse precomputed document caches without per-model stores, retaining about 96% of direct-context F1.","Translator training scales with the number of model pairs rather than requiring a shared latent space for all models, avoiding quadratic alignment costs.","Token-level routing implies that different tokens' cache states benefit from different translation maps, so input-dependent translator selection matters more than simply adding more translators."],"fun_headline_variants":["LLM cache translation keeps 96% quality via translator mix","Heterogeneous LLMs share caches: 96% answer quality preserved","Mixture-of-translators maps KV caches across LLMs, saves memory","Cache translator mixture preserves 96% quality in long-context","One cache, any LLM: MoT retains 96% answer quality"],"cache_read_input_tokens":36992,"weakest_assumption_plain":"The load-bearing premise is that layers at similar relative depths in different models encode corresponding information, so a depth-ratio mapping between source and target layers yields a usable translation channel; the paper's own appendix concedes this weakens across instruction tuning, data distribution, or representation differences.","fun_headline_variants_meta":{"raw":{"variants":["LLM cache translation keeps 96% quality via translator mix","Heterogeneous LLMs share caches: 96% answer quality preserved","Mixture-of-translators maps KV caches across LLMs, saves memory","Cache translator mixture preserves 96% quality in long-context","One cache, any LLM: MoT retains 96% answer quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1229,"prompt_tokens":795,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":352}},"tokens_in":539,"tokens_out":434,"duration_ms":4378,"temperature":1.0,"reasoning_tokens":352,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T16:08:56.528549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a pair of models from genuinely different architecture families with different tokenizers, translate a context cache with the same MoT pipeline, and measure closed-set QA accuracy against direct prefill; if accuracy collapses to near random, the cross-model generality of the claim fails.","supporting_citations":[],"review_version":1}