{"id":"2b321b9e-294d-476b-ad62-fc0f60c8b759","arxiv_id":"2607.08399","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A learned weighted sum of intermediate-layer activations compresses an instruction prompt into a single patch vector that, injected at an early layer, recovers task accuracy within ~2% of the full prompt.","lead":"Instruction prompts in LLMs can be compressed into one activation vector via a learned weighted sum of mid-layer states and reinjected early, recovering nearly full accuracy. This amortizes repeated system prompts and exposes cross-layer structure in how models encode task information.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The under-2% claim is tightly tied to short, fixed-template knowledge tasks; the paper does not show that mid-layer activations remain linearly recoverable for longer or more compositional prompts.","rationale":"The Reader correctly isolates the same soft spot: the under-2% result and the mid-to-early transfer pattern are demonstrated only inside a narrow short-prompt regime, with layer choice justified by a coarse 1-epoch heatmap on that same regime. No internal contradiction or circular derivation appears; the W-MLP vs full/masked comparisons, the TC overfitting contrast, and the weight-attribution diagnostics are coherent evidence for what was actually tested. The concern is therefore one of scope, not of soundness inside the reported experiments. Because the Reader already issued CONDITIONAL precisely for this reason (and for missing public artifacts / error bars), the stress-test does not move the verdict. A single longer-context hold-out experiment would settle whether the load-bearing recoverability assumption generalizes or remains confined to the short knowledge/MCQ setting.","tokens_in":17095,"tokens_out":652,"duration_ms":6811,"concrete_test":"Hold out a longer-context suite (e.g., 200–500-token system prompts or multi-hop ARC/MMLU-style questions) never seen in training. Train W-MLP only on the original short Toy Tasks, extract at layer 12 / inject at layer 2, and measure exact-match or letter accuracy against full-prompt and masked baselines on the long suite. If the gap to full-prompt exceeds ~10 points (or falls near masked), the under-2% claim is regime-specific and the mid-to-early linear recoverability assumption does not hold more generally.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract and §3.2) is that a learned weighted sum of mid-layer activations, injected early, compresses task-relevant prompt information with under 2% accuracy drop relative to full-prompt processing. That number is measured only on the Toy Task suite of short instruction templates (capitals, antonyms, etc.) where W-MLP reaches 85.35% vs full-prompt 86.92% (Table 2). On ARC-Easy the same method already drops ~15 points (77% vs 92% on Llama-3.1-8B, Figure 4), and OOD toy tasks drop further (63%). The layer pair (m=12, e=2) is selected from a 1-epoch heatmap on a subset of the same short tasks (§3.4, Figure 5). The Limitations section itself flags that fidelity under high information-density or longer prompts is open. Thus the load-bearing assumption is that the information needed for the query is sufficiently concentrated in one mid-layer sequence and remains linearly recoverable after early injection outside the short knowledge-retrieval / MCQ regime actually tested. If that concentration fails for longer or multi-hop prompts, the headline accuracy claim does not transfer and the practical reuse story weakens.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes compressing fixed instruction prompts into a single activation-space 'patch' vector via a learned weighted sum of mid-layer hidden states (Weighting MLP), then re-injecting that vector at an early layer through a placeholder token so the frozen LLM can answer queries without the original token sequence. A more expressive Transformer Compressor baseline is also trained. On short instruction-style Toy Tasks with Llama-3.1-8B-Instruct, W-MLP recovers nearly full-prompt accuracy (85.35% vs 86.92% test; Table 2), while ARC-Easy and OOD tasks show larger gaps. Ablations identify mid-layer extraction with early-layer injection as best, and qualitative analyses show the learned weights concentrate on semantically salient tokens. The authors argue this reveals cross-layer compatibility and linearity of task information in activation space, with potential reuse benefits for fixed system prompts.","tokens_in":17428,"tokens_out":1530,"duration_ms":19128,"significance":"If the result holds beyond the evaluated regime, the work offers a lightweight, no-LLM-finetuning route to amortize fixed instruction prefixes in activation space, complementary to KV/prefix caching and distinct from token-level compressors (Gisting, ICAE) that require heavy model-specific training. The finding that a simple weighted sum outperforms an end-to-end Transformer compressor, together with the mid-extract/early-inject pattern and interpretable token weights, is a concrete contribution to activation engineering and the linearity/superposition literature. Strengths include a clear two-pass framework, multi-model checks (including 1B–8B scales), layer/placeholder ablations, a multi-task capacity probe in Appendix C, and promised code/Toy Task release. The practical efficiency story and the structural claims about activation space would be more compelling with broader prompt regimes and quantified compute savings.","major_comments":[{"comment":"Abstract and §3.2 headline: the claim of an accuracy drop 'under 2% relative to full prompt processing' is supported only for in-distribution Toy Tasks (Table 2: 85.35% vs 86.92%). On ARC-Easy the same W-MLP drops ~15 points on Llama-3.1-8B (Figure 4: 77% vs 92%), and OOD Toy Tasks fall to 63.01% vs 94.95% full-prompt. The abstract should qualify the regime (short fixed-template knowledge instructions) or report the full range of gaps; otherwise the central empirical claim overstates what the experiments establish.","section":"Abstract; §3.2 Table 2; Figure 4"},{"comment":"The load-bearing assumption that mid-layer activations remain linearly recoverable after early injection is tested almost exclusively on short knowledge-retrieval and MCQ prompts (Limitations; §3.1). Layer pair (m=12, e=2) is chosen from a 1-epoch heatmap on a Toy Task subset (Figure 5, §3.4). Without at least one longer-context or multi-hop setting, or a stress test where information density is deliberately increased, it is unclear whether the compression mechanism and the practical reuse story transfer outside the current regime. A targeted experiment or a sharply narrowed claim is needed.","section":"§3.4 Figure 5; Limitations"},{"comment":"Tables 2–results and Figure 4 report point accuracies with no error bars, multiple random seeds, or variance across prompt templates. Given that free parameters include extraction/injection layers, W-MLP architecture, and training hyperparameters (Appendix E), and that TC shows clear train/test overfitting, statistical reliability of the <2% gap and of the W-MLP vs TC comparison cannot be assessed. At minimum, multi-seed means and standard deviations on the main Toy Task and ARC-Easy splits should be reported.","section":"Table 2; Figure 4; Appendix E"},{"comment":"Practical implications (§5) assert reduced per-query computation for fixed instruction prompts, yet the method still requires a partial forward pass through roughly half the layers at compression time (Limitations) and no wall-clock, FLOP, or latency comparison against KV/prefix caching is given. Without a quantitative efficiency comparison on a realistic reuse workload, the engineering motivation remains speculative relative to existing exact-reuse mechanisms cited in the introduction.","section":"§1; §5; Limitations"}],"minor_comments":[{"comment":"Figure 3 caption and legend use 'Antonymes' (French spelling); standardize to 'Antonyms' for consistency with the rest of the paper.","section":"Figure 3"},{"comment":"In §2.1 the W-MLP is described as having hidden dimensions [2048, 1024, 512, 256], while Appendix E Table 9 lists 4096→2048→…→1; reconcile the architecture description.","section":"§2.1; Appendix E Table 9"},{"comment":"Placeholder is rendered as '¿' in places and described as U+FFFD; ensure consistent rendering and that the token identity is unambiguous for reproduction.","section":"§3.1; Figure 6"},{"comment":"Related Work could more explicitly contrast sequence-length and attention-cost implications of activation-level vs token-level compression (Gisting, AutoCompressors, ICAE) in one comparative paragraph.","section":"§6"},{"comment":"Appendix A hand-made experiment uses λ=100,000 replacement scaling on the base (non-Instruct) model and reports ~5.9% on Instruct; a brief note on why scaling differs so sharply between base and Instruct would help readers.","section":"Appendix A"},{"comment":"Typos/style: 'W eighting MLP' / 'T ransformer Compressor' appear with odd spacing in headings (§2.1–2.2); 'F uture Extensions' in §5; fix for camera-ready.","section":"§2; §5"}],"recommendation":"major_revision","confidential_remarks":"The technical idea is interesting and the W-MLP vs TC contrast is a genuine positive finding, but the abstract currently sells a near-lossless result that only holds on the easiest split. I would not accept without a rewritten abstract and multi-seed numbers. Scope is a reasonable fit for a methods/interpretability venue; borderline for a top general ML venue unless longer-context evidence is added. No integrity concerns; citation of authors' prior steering work is minor and disclosed."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful result here is simple and concrete: freeze an off-the-shelf LLM, extract mid-layer hidden states from a fixed instruction prompt, compress them with a small MLP that predicts scalar weights for a linear sum, and overwrite a placeholder residual at an early layer. On the Toy Task suite that number is real—W-MLP 85.35% vs full-prompt 86.92% (Table 2)—and the same compressor stays close to the full-prompt ceiling on ARC-Easy for the 8B models (Figure 4). The mid-to-early transfer pattern (Figure 5) and the fact that the simple weighted sum beats the end-to-end Transformer compressor are the two observations I would actually remember.\n\nWhat the paper does well is the experimental hygiene around that claim. Masked vs full baselines bracket the gap, layer and placeholder ablations are reported, the TC overfitting diagnosis is honest, and the weight visualizations (Table 3, Appendix D) show the MLP focusing on the same semantic tokens a hand-crafted rule would pick. Code and the Toy Task set are promised; the training setup is ordinary cross-entropy against frozen logits, so there is no circular derivation.\n\nSoft spots are real but proportionate. The under-2% headline is measured only on short, fixed-template knowledge tasks; ARC-Easy already costs ~15 points and OOD toy tasks drop to 63%. Layer choice comes from a one-epoch heatmap on the same short suite. Limitations correctly flags longer or high-density prompts as open, and there are no error bars or multi-seed numbers. Practical talk about RAG or VLA amortization is therefore still extrapolation. None of that breaks the core technical demonstration.\n\nThis is for people who care about activation engineering, serving costs of repeated system prompts, or linear structure in residual streams. It is not a general long-context compressor yet. I would send it to referees: the result is new enough, the evidence is clean enough inside its regime, and the soft spots are the kind that revision and longer-context tests can address. Worth engaging.","headline":"Clean empirical demo that a lightweight weighted-sum of mid-layer activations, injected early, recovers near full-prompt accuracy on short instruction and ARC-Easy tasks; the under-2% number is real but tightly scoped.","tokens_in":18062,"tokens_out":524,"would_cite":true,"duration_ms":10530,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A single learned activation vector can replace a full instruction prompt, with under 2% accuracy loss versus processing every token.","keywords":["prompt compression","activation aggregation","weighted sum","activation patching","LLM efficiency","representation engineering","cross-layer transfer"],"falsifier":"Train and evaluate the same weighted-sum patch setup on longer multi-hop or dense reasoning prompts (for example multi-paragraph instructions or harder multi-choice suites); if accuracy collapses far below the full-prompt baseline while a masked placeholder stays near chance, a single patch vector does not preserve task-relevant information beyond the short-prompt regime.","tokens_in":17967,"feed_emoji":"🗜️","tokens_out":852,"duration_ms":19212,"temperature":0.7,"pith_summary":"Large language models often recompute the same fixed instruction or system prompt on every query. This paper asks whether the useful information in such a prompt can be squeezed into one activation vector and plugged back into the model instead of reprocessing the tokens. The authors train a small network to weight mid-layer activations, sum them into a patch vector, and overwrite a placeholder token’s early-layer activation so the model answers without seeing the original prompt. On controlled knowledge tasks and ARC-Easy questions, this recovers nearly the full-prompt accuracy—within about 2%. The result also points to structure in activation space: mid-layer states remain usable when injected early, a single vector can carry recoverable semantic content, and a weighted sum is enough to compress it.","feed_headline":"One activation vector replaces a full prompt with under 2% loss","feed_subtitle":"A weighted sum of mid-layer states, re-injected early, recovers nearly full instruction accuracy.","key_machinery":"The Weighting MLP (W-MLP): a small feed-forward net that maps each mid-layer token activation to a scalar weight; the patch vector is the weighted sum of those activations. That vector replaces the hidden state of a neutral placeholder at an early layer so the frozen LLM continues its forward pass without the original prompt tokens.","core_discovery":"Task-relevant information from an instruction prompt can be compressed into a single activation vector by a learned weighted sum of hidden states taken from an intermediate layer, then re-injected by overwriting a placeholder token’s activation at an early layer, keeping accuracy within roughly 2% of full prompt processing on the evaluated tasks.","pith_inferences":["Amortized over many queries with the same instruction, this could cut compute more aggressively than KV-caching alone because the sequence length itself shrinks at inference.","The same patch vectors could double as compact semantic keys for retrieval systems that share a representation between search and generation.","Segmenting a long prompt into several patch tokens (one per semantic block) is a natural next step to raise fidelity without leaving activation space.","The learned token weights give a built-in importance signal that could be compared directly to post-hoc feature-attribution scores."],"forward_implications":["Fixed system prompts can be pre-compressed once and reused without re-encoding the original token sequence on every query.","Mid-layer representations transfer meaningfully into early layers, indicating cross-layer compatibility of how information is encoded.","A single activation vector can encode a quantifiable and recoverable amount of semantic task information.","A simple weighted sum of activations can serve as a robust compressor and can outperform a more expressive end-to-end transformer compressor on the same task.","Compression quality appears to improve as the model’s hidden dimension grows."],"fun_headline_variants":["One mid-layer activation vector replaces full prompts under 2% loss","Learned weighted activations compress prompts into a single vector","Re-inject early a weighted mid-layer sum to skip full prompt tokens","Task info packs into one activation vector with under 2% accuracy drop","Cross-layer activation transfer compresses fixed prompts nearly intact"],"cache_read_input_tokens":2432,"weakest_assumption_plain":"Everything the model needs from the prompt is concentrated enough in one mid-layer activation sequence to survive compression into a single vector and early-layer injection—shown so far only on short knowledge and multiple-choice prompts.","fun_headline_variants_meta":{"raw":{"variants":["One mid-layer activation vector replaces full prompts under 2% loss","Learned weighted activations compress prompts into a single vector","Re-inject early a weighted mid-layer sum to skip full prompt tokens","Task info packs into one activation vector with under 2% accuracy drop","Cross-layer activation transfer compresses fixed prompts nearly intact"]},"model":"grok-4.5","effort":"low","cost_usd":0.005268,"raw_usage":{"total_tokens":1410,"prompt_tokens":702,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":52680000,"prompt_tokens_details":{"text_tokens":702,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":616,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":702,"tokens_out":92,"duration_ms":6018,"temperature":1.0,"reasoning_tokens":616,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T08:02:28.998620+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train and evaluate the same weighted-sum patch setup on longer multi-hop or dense reasoning prompts (for example multi-paragraph instructions or harder multi-choice suites); if accuracy collapses far below the full-prompt baseline while a masked placeholder stays near chance, a single patch vector does not preserve task-relevant information beyond the short-prompt regime.","supporting_citations":[],"review_version":1}