{"id":"4fc50c6b-3f06-4d05-9f46-69f75d0f370c","arxiv_id":"2607.27090","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"GPU-resident precomputed KV injection with Chunked RoPE and context-window encoding makes personalized LLM memory latency nearly independent of retrieval budget while nearly matching prompt-injection accuracy.","lead":"InferScale precomputes each memory fact’s key–value state on the GPU and injects it into vLLM’s cache instead of re-prefilling retrieved text every request. That keeps time-to-first-token nearly flat as more memory is retrieved, cutting latency several-fold versus production prompt-injection memory systems.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The paper's central claim is a systems result: KV injection + Chunked RoPE makes memory-conditioned TTFT nearly independent of retrieval budget, with quality close enough for the application. Exactness is proved where claimed (Thm 3 for joint encoding at fixed positions; Thm 6 for pre-RoPE relocation); the only approximation is independent fact encoding, which the authors surface, mitigate offline with windows, and quantify (w=0 collapses; w≥20 recovers to 60.3% vs 63.3%; multi-hop is the residual). That matches the reader's weakest_assumption exactly. Because the latency/throughput evidence does not rest on closing that gap, and the paper does not over-claim parity on multi-hop or exhaustiveness vs every KV-reuse baseline, the concern is already priced into a sound ACCEPT. No stronger load-bearing flaw (e.g., incorrect RoPE handling, confounded baseline, or theory–system mismatch) appears on a full-manuscript read. Verdict stays ACCEPT / HIGH; no adjustment.","tokens_in":29881,"tokens_out":563,"duration_ms":11015,"concrete_test":"Re-run the Llama-3.1-8B LoCoMo accuracy sweep at k=50 with an oracle encoding window that includes every other fact that ends up in the same retrieved set (approximating Corollary 9's full-prefix case) and compare multi-hop vs overall accuracy to Mem0; if the residual gap closes to ~0 while TTFT remains flat, the approximation is confirmed as the sole quality limiter and the systems claim is unchanged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (finite context windows only approximate joint cross-fact prefill, with residual multi-hop gap) is real but already scoped correctly by the paper and does not undermine the central systems claim. Theorems 3 and 6 establish exact equivalence for jointly encoded memory and exact position relocation under pre-RoPE storage; Context-Window Encoding (Theorem 8, Corollary 9) is explicitly an approximation whose cost is measured (Table 2, Appendix B). Latency/throughput results (flat TTFT, 3.6–4.8× at k=50, 3.7–4.5× QPS) are independent of that approximation and are the load-bearing evidence for the stated claim that reusable KV decouples serving latency from retrieved-context size while preserving application quality within a few points. No hidden inconsistency, unacknowledged assumption failure, or measurement confound rises to a load-bearing concern against ACCEPT.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"InferScale replaces repeated prompt prefilling of retrieved personalized memory with reusable, GPU-resident KV state. Memory facts are encoded offline (with optional preceding-turn context), stored pre-RoPE alongside semantic embeddings, retrieved at serving time via a GPU ANN index (Jasper), re-rotated with Chunked RoPE, and scatter-copied into vLLM’s paged KV cache through the stock KV-connector interface—no engine changes or fine-tuning. Theorem 3 proves exact hidden-state/output equivalence of KV vs. prompt injection at query positions for jointly encoded memory under causal decoders; Theorem 6 shows pre-RoPE storage plus on-injection rotation yields exact position relocation. Context-Window Encoding (Theorem 8) is the explicit approximation that recovers most cross-fact context. On LoCoMo across Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B (plus Qwen3-14B), engine TTFT stays nearly flat from k=5 to k=50 (e.g., 16.6→17.3 ms on Llama), 3.6–4.8× lower than Mem0 at k=50, with 60.3% vs. 63.3% accuracy at w=50 and 3.7–4.5× throughput at 100 users; CPU offload of the KV store adds only a few ms to TTFT.","tokens_in":29982,"tokens_out":1287,"duration_ms":34931,"significance":"The central systems claim—that reusable KV state decouples memory-conditioned TTFT from retrieved-context size while preserving application quality within a few points—is well supported and practically important for production personalized serving. Strengths include: (i) clean, correctly scoped theory (exact joint-encode equivalence and exact RoPE relocation, with the independent-fact residual explicitly measured rather than papered over); (ii) a non-invasive vLLM plugin implementation; (iii) thorough ablations of k, w, GPU vs. CPU index, GPU-resident vs. offloaded KV, model scale to 14B, and bf16 empirical equivalence; and (iv) clear separation of offline encoding cost from serving latency. If the result holds under broader workloads, attention-layer injection of static retrieved context is a natural serving primitive beyond Mem0-style memory systems.","major_comments":[{"comment":"§6.3, Table 2 and Appendix B: the residual multi-hop gap versus Mem0 (up to ~16 points at k=50 on Mistral) is the load-bearing accuracy caveat of independent encoding. Corollary 9 states exact joint-prefill KV only when the encoding window contains the entire retrieved prefix, which selective top-k does not guarantee. The paper already scopes this as an approximation, but the main claim’s “preserving application quality” language should be tightened in the abstract/conclusion to state that quality is recovered to within a few overall points, with the multi-hop shortfall remaining the principal residual—and ideally add a short analysis of how often retrieved facts at serving time were co-present in each other’s offline windows.","section":"§6.3, Table 2, Appendix B, Corollary 9"},{"comment":"§6.4 throughput setup: all N concurrent users share one conversation’s memory corpus and issue queries against that single store. This stresses batching and prefill elimination but does not exercise per-user KV isolation, multi-tenant HBM pressure, or cross-user retrieval contention that production personalized serving would see. A brief multi-corpus or multi-store measurement (even at smaller N), or an explicit limitation statement, would make the 3.7–4.5× QPS claim more transferable.","section":"§6.4"}],"minor_comments":[{"comment":"Figure 4 caption and §6.2: state explicitly that prefix caching is enabled for Mem0 (best case) and that InferScale curves for w∈{0,5,20,50} overlap; the main text does, but a one-line callout in the figure helps.","section":"Figure 4, §6.2"},{"comment":"Theorem 3 title has a typo: “Eqivalence” → “Equivalence” (also Corollary 4).","section":"§4.2 Theorem 3, Corollary 4"},{"comment":"§6.9 / Table 7: one Qwen open-ended case is “Different” under greedy bf16; a sentence on whether the factual content still matched (table says 100% factual accuracy) would close the loop for readers worried about non-determinism.","section":"§6.9, Table 7"},{"comment":"Related work (§1 closing, §2): CacheBlend, LMCache, Block-Attention, LazyAttention are contrasted at a high level; a short table of assumptions (fixed layout vs. dynamic assembly; fine-tuning required; recompute) would sharpen the positioning.","section":"§1, §2"},{"comment":"Table 5 / §6.7: report KV footprint also per fact or per 1K tokens so readers can extrapolate beyond LoCoMo conversation sizes.","section":"§6.7, Table 5"},{"comment":"Minor notation: m is used both for memory token length and (implicitly) for composed segment length; a single consistent definition early in §4 would help.","section":"§4"}],"recommendation":"minor_revision","confidential_remarks":"I agree with the reader’s ACCEPT lean on technical soundness; I am recommending minor_revision only to force a tighter accuracy claim and a clearer throughput-setup caveat before camera-ready. No integrity or novelty concerns. Fit for a systems/DC venue is strong."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a real systems answer to repeated prefill of retrieved user memory. They keep fact KV on the GPU, inject it into vLLM’s paged cache, and stop TTFT from scaling with k. That claim is measured cleanly against Mem0 on LoCoMo across three 7–8B models, plus a 14B check.\n\nWhat is new is not “reuse KV” in the abstract—prefix caches and CacheBlend already do fixed layouts—but the co-design for dynamically assembled, position-varying memory: store keys pre-RoPE, re-rotate at injection (Chunked RoPE), and encode each fact behind a small offline context window while caching only the fact’s slice. Theorems 3 and 6 are standard causal/RoPE algebra, correctly scoped: joint encode is exact at query positions; independent facts are the approximation. The plugin path (no engine fork, no fine-tune) is the right engineering move.\n\nWhat they do well: flat engine TTFT (~16–18 ms from k=5 to 50 vs Mem0 roughly doubling), 3.6–4.8× at k=50, 3.7–4.5× QPS at 100 users, CPU offload that only adds a few ms to TTFT, and honest accuracy tables. w=0 collapses as k grows; w≥20 mostly recovers overall accuracy (60.3% vs 63.3% on Llama at k=50). Appendix multi-hop gap is real and they own it. Empirical bf16 equivalence and order-independence checks support the theory in practice. Citations to Mem0/MemGPT/Zep and the KV-reuse line are fair.\n\nSoft spots, in proportion: residual multi-hop lag is the main quality cost of independent encoding—Corollary 9 only gets exact joint KV if the window holds the whole retrieved prefix, which top-k does not guarantee. Baselines vs other dynamic KV-reuse systems are thinner than vs Mem0. GPU KV footprint (1.8–4.8 GB/conversation) is the capacity story; offload helps but is PCIe-bound end-to-end. None of that breaks the latency decoupling claim.\n\nFor people building agent memory or personalized serving, this is worth reading and citing. I would bring it to reading group, send it to peer review, and engage with the code path. Recommend accept-with-normal-revision pressure on multi-hop analysis and a couple more KV-reuse baselines—not a desk reject.","headline":"Solid MLSys co-design: pre-RoPE KV injection plus Chunked RoPE actually flattens TTFT for dynamic personal memory, with the multi-hop accuracy gap scoped rather than hidden.","tokens_in":30709,"tokens_out":628,"would_cite":true,"duration_ms":14654,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Reusable KV state makes personalized LLM serving latency nearly independent of how much memory is retrieved.","keywords":["KV cache injection","LLM serving","personalized memory","rotary position embeddings","time-to-first-token","GPU-native retrieval","vLLM","context-window encoding"],"falsifier":"On LoCoMo-style multi-hop questions at large k, measure whether context-window KV injection closes the accuracy gap to joint prompt injection; a persistent large multi-hop deficit as windows grow would falsify the claim that quality is preserved.","tokens_in":30638,"feed_emoji":"⚡","tokens_out":953,"duration_ms":15951,"temperature":0.7,"pith_summary":"Production memory systems retrieve user facts and paste them into the prompt, so the model re-prefills the same text on every request and time-to-first-token grows with the retrieval budget. InferScale instead precomputes each fact’s key–value attention state once, keeps it on the GPU next to a semantic index, and at serving time injects that KV straight into the engine’s paged cache so only the query is prefilled. Chunked RoPE stores keys before rotary position encoding and rotates them at injection, so a fact encoded once can sit at any prompt position. Context-window encoding runs each fact behind a short offline window of prior turns but caches only the fact’s KV, recovering most of the accuracy that independent encoding would lose. On LoCoMo across three open-weight models, engine TTFT stays nearly flat from k=5 to k=50 while a prompt-injection baseline more than doubles, yielding multi-fold latency and throughput gains with accuracy close to full prompt injection.","feed_headline":"KV injection flattens memory-serving latency as retrieval grows","feed_subtitle":"Precomputed attention state cuts TTFT 3.6–4.8× at k=50 while holding accuracy near prompt injection","key_machinery":"KV injection with Chunked RoPE: store each fact’s keys before rotary position encoding, re-rotate them to serving-time positions on injection into vLLM’s paged cache, and prove exact attention equivalence to prompt injection (Theorems 3 and 6). Context-Window Encoding supplies the residual accuracy by conditioning each fact offline on a window of preceding turns while caching only the target fact’s KV.","core_discovery":"For causal decoders, injecting precomputed memory KV at the intended positions is exactly equivalent to prompt-injecting the same tokens at query positions; with keys stored pre-RoPE, that equivalence survives dynamic reassembly. Therefore reusable KV state can decouple memory-conditioned serving latency from retrieved-context size while context-window encoding keeps application quality near the prompt-injection baseline.","pith_inferences":["If multi-hop remains the residual gap, hybrid schemes that jointly re-encode only the retrieved clique at serving time could close accuracy without returning to full prefill cost.","Online insert/revise/delete of facts would turn the offline KV store into a live memory OS; consistency between the ANN index and pre-RoPE tensors becomes the hard systems problem.","Paging hot facts in HBM and cold facts in host/disk along the retrieval path is the natural next capacity layer once per-conversation KV reaches multi-GB."],"forward_implications":["Memory-conditioned TTFT need not grow with retrieval budget k if facts are injected as KV rather than re-prefilled as text.","A fact encoded once can be placed at arbitrary prompt positions under RoPE without re-encoding, enabling dynamic top-k assembly.","GPU-resident retrieval plus KV injection multiplies concurrent-user throughput because prefill work shrinks to the query alone.","Offloading the pre-RoPE KV store to host DRAM lifts per-GPU user capacity at only milliseconds of extra end-to-end latency.","The same retrieve-and-inject primitive applies to any largely static retrieved context (stable RAG corpora, system prompts, tool descriptions), not only personal memory."],"fun_headline_variants":["Precomputed KV flattens TTFT as memory retrieval grows","GPU-native KV injection cuts memory TTFT 3.6–4.8× at k=50","Reusable memory KV keeps TTFT flat while accuracy stays near baseline","Chunked RoPE lets InferScale inject dynamic memory without reprefilling","InferScale decouples serving latency from retrieved memory size"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That encoding each fact with only a finite offline window of prior turns recovers enough cross-fact context for answer quality when retrieval later assembles those facts in a different order and set than the encoding windows saw.","fun_headline_variants_meta":{"raw":{"variants":["Precomputed KV flattens TTFT as memory retrieval grows","GPU-native KV injection cuts memory TTFT 3.6–4.8× at k=50","Reusable memory KV keeps TTFT flat while accuracy stays near baseline","Chunked RoPE lets InferScale inject dynamic memory without reprefilling","InferScale decouples serving latency from retrieved memory size"]},"model":"grok-4.5","effort":"low","cost_usd":0.002013,"raw_usage":{"total_tokens":1018,"prompt_tokens":915,"num_sources_used":0,"completion_tokens":83,"cost_in_usd_ticks":20128000,"prompt_tokens_details":{"text_tokens":915,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":20,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":915,"tokens_out":83,"duration_ms":2175,"temperature":1.0,"reasoning_tokens":20,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:27:13.266990+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On LoCoMo-style multi-hop questions at large k, measure whether context-window KV injection closes the accuracy gap to joint prompt injection; a persistent large multi-hop deficit as windows grow would falsify the claim that quality is preserved.","supporting_citations":[],"review_version":1}