{"id":"9f2a8738-15fc-41ee-8a0a-b79ed8ddba53","arxiv_id":"2603.18908","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Linear (affine) maps between final hidden states of independent LLMs preserve downstream performance and can enable text generation when tokenizers and scale align, enabling a practical HE-based privacy protocol.","lead":"Affine maps between final hidden states of independently trained LLMs largely preserve classification, OOD detection, and sometimes enable coherent cross-model text generation. The work also builds HELIX, a privacy framework that encrypts only the linear head for sub-second cross-silo inference.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Generation success may be overstated as true zero-shot cross-model transfer because token-offset alignment and ridge fitting on MMLU/Alpaca can leak target-head structure.","rationale":"The Reader correctly flags the public-data assumption for HELIX and the need for clearer scope language, but the more central risk to the paper's novelty claim is the generation protocol itself. Classification and OOD transfer (Table 1) remain solid under the reported linear maps; the generation results, however, rest on a token-alignment and fitting procedure that can inject target-head information before any decoding occurs. The proposed test isolates that confound without requiring new models. If the test holds, the claim is strengthened and the Reader's CONDITIONAL verdict is unchanged; if it fails, the abstract's \"for the first time\" generation statement must be qualified, keeping the overall verdict CONDITIONAL but for a sharper reason. Code release remains necessary either way.","tokens_in":35440,"tokens_out":581,"duration_ms":6212,"concrete_test":"Retrain every high-quality pair in Table 3 / Fig. 5 using only a pure public corpus (Wikipedia/IMDB) with no MMLU/Alpaca tokens and with deliberately mismatched offsets (random or source-only BPE boundaries). Re-evaluate LLM-judge scores, embedding similarity, and MMLU letter accuracy. If scores for pairs with exact-match >0.7 fall below 3.0 or MMLU accuracy collapses >15 pp relative to the reported mapped numbers, the zero-shot generation claim is overstated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest claim is that a pure affine map on final/penultimate states enables coherent zero-shot text generation across independently trained models (abstract; §3.2; Fig. 1). The map is fit by OLS ridge (λ=10^{-4}) on 4k examples from MMLU or Alpaca, with token pairs produced by character-offset matching: j=arg min_k {e_k^B | e_k^B ≥ e_i^A} (§3.2). That procedure is not architecture- or vocabulary-agnostic; it supplies the source with target-side token boundaries and trains W* so that mapped states are already close to the frozen target LM head's preferred inputs. Consequently the high-quality pairs (exact-match ≥0.67, r=0.898 with judge scores) may succeed largely because the alignment already encodes target-head geometry rather than because late residual streams are freely interchangeable. Classification/OOD results (Table 1) are less affected, but the generation novelty claim is the load-bearing one.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper empirically characterizes how well affine maps between final (or penultimate) hidden states of independently trained LLMs preserve downstream behavior. It shows that a fixed target linear head largely retains classification accuracy and Energy-based OOD AUROC when applied to source embeddings after a learned map (Tables 1–2, 6–7). For instruction-tuned models it further reports that the same style of map, trained by ridge regression on character-offset-aligned token pairs, can drive the frozen target LM head to produce coherent text for some pairs (Figs. 1, 3–5; Tables 3–4), with success strongly predicted by tokenizer exact-match rate (r=0.898, n=23) and source scale (≥4B). Building on late-layer compatibility, it proposes HELIX: a semi-honest two-party protocol that fits the map on public (optionally few-shot ID) data and encrypts only the linear classification step under CKKS, claiming sub-second latency and <1MB communication.","tokens_in":35690,"tokens_out":1505,"duration_ms":34280,"significance":"If the generation results hold under the stated conditions, this is a useful empirical extension of model stitching and linear identifiability to open instruction-tuned LLMs, with a clear practical takeaway (tokenizer overlap and source scale as predictors) and multi-signal evaluation (embedding similarity, dual LLM judges, human ratings on 200 samples, layer-depth ablation). The classification/OOD transfer results strengthen the case that vendor and open embedding spaces share usable linear structure. HELIX is a sensible systems application of that observation: encrypting only the linear head is a real efficiency win relative to full-transformer HE/MPC baselines. Strengths include breadth of model pairs, explicit tokenizer analysis, Appendix E’s depth sweep localizing alignability to late residual-stream states, and a concrete privacy protocol with threat-model discussion. The work is primarily empirical rather than theoretical; its value is in careful measurement and a deployable sketch, not a new theorem.","major_comments":[{"comment":"Abstract and §3.2 claim that linear alignment “sometimes enables text generation” and is evaluated as zero-shot cross-model decoding. The procedure is standard stitching (map h_B→h_A, freeze f_A), not leakage of the target head weights into W*, but training still uses 4k MMLU/Alpaca pairs with character-offset token matching (j=arg min_k {e_k^B | e_k^B ≥ e_i^A}). Combined with the load-bearing finding that exact token match rate predicts judge scores (r=0.898, Fig. 5; high-quality pairs ≥0.67 match), the result is better stated as: late residual streams are linearly interchangeable for multi-step decoding primarily when tokenizers already largely agree and the source is large enough. Please elevate that scope condition into the abstract and intro so the novelty claim is not read as unrestricted residual-stream interchangeability.","section":null},{"comment":"HELIX’s pure public-data setting (§4–5, Eq. 1, Table 2) is central to the privacy story, yet Public Only often collapses relative to Full Data (e.g., TREC Gemini→OpenAI 57.6% vs 95.4%; several MNLI rows near or below useful baselines). Public+ID (64–128) recovers much of the gap but, as the paper notes, weakens zero-shot confidentiality. The abstract’s “shared public dataset” framing and the sub-second latency claim should be paired with a clear, prominent statement of when public-only transfer is adequate versus when few-shot ID exposure is required; otherwise the application overstates utility under the strongest privacy constraint.","section":null},{"comment":"§3.2.1 / left of Fig. 3: asymmetric MMLU transfer (strong→weak preserves accuracy; weak→strong degrades sharply) is important for interpreting what is being transferred. The text attributes this to source capacity, but without controls that hold tokenizer fixed while varying scale (or vice versa) the confound with tokenizer family and training recipe remains. A short controlled comparison (same-family scale pairs already partially present, e.g. Llama-3.2-1B→Llama-3-8B) should be discussed explicitly as evidence for the capacity claim rather than left as a qualitative pattern.","section":null}],"minor_comments":[{"comment":"Fig. 1 caption and hybrid-model description are clear; ensure the main text states unambiguously that only the affine map is learned and both transformer stacks remain frozen.","section":null},{"comment":"§3.2: “penultimate layer” vs “final hidden states” / post-final-norm is clarified in Appendix E; a one-sentence pointer in the main generation setup would help readers who skip the appendix.","section":null},{"comment":"Table 1 OOD columns sometimes improve over baseline (e.g., TREC OpenAI→Qwen 0.738→0.916). Briefly discuss whether this is calibration recovery, distribution shift of Energy scores, or noise from small OOD sets.","section":null},{"comment":"Related work: distinguish more sharply from Chen et al. (2025) (linear feature transfer for efficiency) and Jiang & Li (2024) (GPT–BERT stitching) so the generation-under-frozen-LM-head claim is the explicit delta.","section":null},{"comment":"Typo/fragment in §3.2 Experimental Setup: “give me this in latex: We extract hidden states…” appears to be an author note left in the manuscript; remove before camera-ready.","section":null},{"comment":"Appendix algorithms: Algorithm 2’s input list says alignment parameters are held by PARTYA, which contradicts the main text (PARTYB retains W*). Align the pseudocode with §4.3 / H.6–H.7.","section":null},{"comment":"Human evaluation (§B.2) uses a single evaluator on 200 samples; state that limitation and the blinding procedure more clearly in the main text.","section":null},{"comment":"Report ridge λ=10^{-4} and 4k training size sensitivity (Fig. 12) is useful; consider a one-line main-text note that results are stable near that operating point.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The generation result is the main novelty relative to concurrent stitching work; it is real for high-compatibility pairs but easy to over-read. I would not reject on the skeptic’s “target-head leakage” framing—the map is fit to target hidden states, not logits—but the abstract should stay tightly scoped. HELIX is a nice systems angle for this venue if the public-only vs few-shot tradeoff is not buried. Overall a solid empirical paper that fits a methods/systems AI venue after tightening claim language."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is that late residual streams of modern LLMs often share enough linear structure that a simple ridge map preserves linear-head classification and OOD energy scores, and—for tokenizer-compatible pairs with source models ≥4B—can even drive coherent greedy generation through a frozen foreign LM head. That generation result is the real increment over Bansal, Chen et al., and the Platonic-representation literature; the HELIX protocol is a clean engineering consequence that gets sub-second latency by encrypting only the linear head.\n\nWhat they do well is the measurement. Classification and OOD tables are consistent across embedding APIs and datasets with proper baselines. Generation is checked three ways (embedding similarity, dual LLM judges, 200-sample human ratings) and they surface the failure modes: exact token-match rate r=0.898 (n=23) and the hard scale floor. The layer sweep is especially clean—alignability is essentially a step function at the final block—so the “late-stage only” claim is not hand-waving. Math is ordinary ridge OLS; no circularity. Citations are appropriate.\n\nSoft spots are real but proportional. The stress-test note is half-right: character-offset token pairing plus fitting on MMLU/Alpaca does give the map target-side boundary information, so “zero-shot architecture-agnostic transfer” overstates it. Still, the map is pure affine, the head is frozen, and same-family perfect-tokenizer small models still fail, so capacity and late geometry matter. Abstract language should match the body (“sometimes,” “tokenizer-compatible”). Human N is modest; code is promised but not yet public. The public-data assumption for HELIX is the usual practical compromise and they quantify the few-shot privacy–utility trade-off.\n\nThis is for people working on representation similarity, model stitching, or privacy-preserving inference who need concrete numbers rather than another CKA heatmap. It deserves a serious referee. I would engage: cite the generation conditions and the HELIX latency numbers, and bring the layer-sweep + tokenizer correlation to reading group.","headline":"Solid empirical map of when linear stitching works for LLMs, including the first clear demos of multi-step generation for compatible pairs, plus a practical HE application; generation novelty is real but narrower than the abstract implies.","tokens_in":36313,"tokens_out":527,"would_cite":true,"duration_ms":6914,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Simple linear maps between final hidden states of independently trained language models largely preserve classification and can sometimes generate coherent text across models.","keywords":["linear alignment","model stitching","representational similarity","cross-model generation","homomorphic encryption","privacy-preserving inference","CKA","language models"],"falsifier":"Train the same affine map on a deliberately mismatched public corpus (different domain and length distribution) and measure whether classification accuracy, OOD AUROC, and Alpaca generation quality collapse relative to an in-distribution map; a large drop would show the public-data assumption is load-bearing.","tokens_in":36305,"feed_emoji":"🔗","tokens_out":962,"duration_ms":12609,"temperature":0.7,"pith_summary":"This paper asks how far independently trained language models have converged in their output representations, and whether that convergence is practical enough to use. The authors learn affine maps that send one model’s final (or penultimate) hidden states into another model’s feature space, then run the target’s frozen linear head or language-model head on the mapped vectors. Across embedding APIs and open models they show that classification accuracy and out-of-distribution detection remain largely intact, and—for the first time—that the same linear stitch can produce coherent zero-shot text generation on some model pairs. Success for generation is strongly predicted by tokenizer compatibility (exact token-match rate r = 0.898) and by source-model scale (pairs with sources below roughly 4B parameters fail). Building on that compatibility they introduce HELIX, a privacy protocol that fits the map on a shared public corpus and encrypts only the linear classification step under homomorphic encryption, yielding sub-second cross-silo inference without sharing private data or full model weights.","feed_headline":"Linear maps let separate LMs share heads—and sometimes talk","feed_subtitle":"Affine stitches preserve classification and can drive coherent generation when tokenizers match","key_machinery":"Affine alignment of final (or penultimate) hidden states: given source embeddings Z_B and target embeddings Z_A on shared inputs, solve W* = (Z_B^T Z_B + λI)^(-1) Z_B^T Z_A (ridge regression) and decode with the target’s frozen head. HELIX encrypts only this linear map and the linear classifier under CKKS, keeping the client’s query private while achieving sub-second latency.","core_discovery":"A learned affine transformation between the final hidden states of independently trained language models is often sufficient to transfer a fixed linear head for classification and out-of-distribution detection with little loss, and in favorable cases is also sufficient to drive coherent autoregressive text generation through another model’s frozen language-model head. Tokenizer exact-match rate and source-model scale are the primary predictors of generation success; the linear correspondence itself is concentrated at the top of the residual stream.","pith_inferences":["If late-layer linear interchangeability is a general consequence of large-scale pretraining, modular multi-vendor pipelines could treat model heads and bodies as separately swappable components.","Tokenizer-agnostic alignment (or shared subword spaces) would be the highest-leverage extension for making generation transfer routine rather than pair-specific.","The same public-map + HE-linear-head pattern could apply to any modality where final representations already show high CKA, not only language embeddings."],"forward_implications":["Cross-model classification and OOD detection can be performed by shipping only a small affine map plus a frozen linear head, without exchanging private data or full model weights.","When tokenizer overlap is high and the source model is large enough (≥4B), zero-shot hybrid generation becomes possible: source embeddings decoded by a different model’s language-model head.","Privacy-preserving cross-silo inference can encrypt only the final linear step, reducing latency from tens of seconds to sub-second under a semi-honest threat model.","Layer sweeps imply that late-stage residual-stream geometry, not intermediate layers, is where cross-model linear interchangeability concentrates."],"fun_headline_variants":["Linear maps transfer heads across separate LMs—and can drive generation","Affine maps let independent LMs share heads for classification and talk","Cross-LM linear alignment preserves heads and enables coherent generation","Final-state affine stitches allow LMs to share heads—and sometimes generate","Independent LMs share fixed heads via linear maps with little performance loss"],"cache_read_input_tokens":32896,"weakest_assumption_plain":"A shared public dataset is distributionally good enough to learn a linear map that carries the target model’s decision boundaries and next-token structure without private data or matching architectures.","fun_headline_variants_meta":{"raw":{"variants":["Linear maps transfer heads across separate LMs—and can drive generation","Affine maps let independent LMs share heads for classification and talk","Cross-LM linear alignment preserves heads and enables coherent generation","Final-state affine stitches allow LMs to share heads—and sometimes generate","Independent LMs share fixed heads via linear maps with little performance loss"]},"model":"grok-4.5","effort":"low","cost_usd":0.005502,"raw_usage":{"total_tokens":1473,"prompt_tokens":738,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":55020000,"prompt_tokens_details":{"text_tokens":738,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":643,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":738,"tokens_out":92,"duration_ms":6085,"temperature":1.0,"reasoning_tokens":643,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T22:19:03.867990+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same affine map on a deliberately mismatched public corpus (different domain and length distribution) and measure whether classification accuracy, OOD AUROC, and Alpaca generation quality collapse relative to an in-distribution map; a large drop would show the public-data assumption is load-bearing.","supporting_citations":[],"review_version":1}