{"id":"8e10a7a9-f298-4a52-b4b0-8d75d01e80cc","arxiv_id":"2607.09153","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"KV-PRM reuses generation KV caches plus a single verify token to score process rewards at O(L) cost, matching text-PRMs with up to 5000\times fewer FLOPs on MATH/GSM8K/AIME.","lead":"KV-PRM scores multi-agent LLM trajectories by reading the generator's existing key-value cache with one verify token instead of re-encoding the full text. This drops scoring cost from quadratic to linear in length while matching or beating text-based process reward models on math benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The empirical superiority claim is load-bearing but rests on a same-architecture, same-data Text-PRM baseline that may understate what a properly optimized text verifier can achieve.","rationale":"The reader correctly flags the shared-architecture constraint and Linear Representation Hypothesis as theoretical soft spots, but those assumptions are not required for the systems result that actually drives the paper: single-token readout over a generation KV cache is O(L) and empirically competitive. The more load-bearing empirical risk is that the accuracy half of the strongest claim is measured only against a deliberately matched Text-PRM that inherits the same model and labels; a stronger or differently architected text baseline is never shown. That does not overturn the complexity or latency numbers (Fig. 3, cost columns), so the verdict stays CONDITIONAL rather than moving to REJECT, but it is the concrete place where the “matches or strictly outperforms” language could fail under a more adversarial baseline. The reader’s weakest-assumption focus is therefore only partially aligned with the claim that most needs stress-testing.","tokens_in":16089,"tokens_out":652,"duration_ms":6956,"concrete_test":"On MATH with Qwen3-8B sequential MAS and the same MCTS trees used for Table 1, train an independent Text-PRM (same LoRA budget) on a different backbone (e.g., Qwen3-4B or a non-Qwen 7–8B) and also a same-backbone Text-PRM with 2–4× more training steps / higher LoRA rank; re-score the identical search trees. If either text variant reaches or exceeds KV-PRM-8B accuracy at still-lower total scoring FLOPs than a same-size Text-PRM, the outperformance claim is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is not merely the O(L) complexity reduction (which follows directly from single-token attention over a pre-existing cache) but that this reduction comes with matching or strictly better accuracy than text-PRMs under Beam Search, MCTS and Weighted Voting (Tables 1–3). That accuracy claim is established only against a Text-PRM that (i) shares the identical base model and LoRA rank, (ii) is trained on the same MCTS Q-value labels, and (iii) re-encodes the full trajectory with the adapter active on every token. Because the generator and verifier must share architecture for KV transfer (§8 limitation), the comparison never tests a stronger, independently trained, or differently sized text verifier that could close the accuracy gap while remaining cheaper than a same-size Text-PRM. The theory (Prop. 1, Thm. 1–2) explains why KV should not be worse, but does not guarantee the observed outperformance once the text baseline is allowed to use a different architecture or more capacity. If a carefully tuned text-PRM of comparable or lower total FLOPs matches KV-PRM accuracy, the “matches or strictly outperforms” half of the strongest claim weakens even though the systems win remains.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes KV-PRM, a process reward model that scores multi-agent trajectories by attending a single verify token to the generator’s pre-existing KV cache (with a LoRA adapter), reducing per-call scoring cost from O(d L^{2}) to O(d L). Under a linear-representation hypothesis and a low-rank reward assumption, it proves that the KV cache is information-theoretically richer than decoded text (Proposition 1, Theorem 1) and that a depth-1 readout already captures most extractable reward information with exponentially diminishing returns for larger k (Theorem 2). Empirically, on MATH, GSM8K, AIME 2024/2025 with Qwen3-0.6B/4B/8B under sequential and hierarchical MAS topologies, KV-PRM matches or exceeds a same-architecture Text-PRM baseline under beam search, MCTS and weighted voting, while reporting up to ~5000× fewer scoring FLOPs, 15–37× lower latency and up to 34.2× lower per-sequence memory. A short KV-Steering proof-of-concept shows that the differentiable KV scores can be used for gradient-based latent message optimization at agent handoffs.","tokens_in":16430,"tokens_out":1215,"duration_ms":11769,"significance":"If the accuracy–efficiency trade-off holds under broader baselines, the work removes a genuine systems bottleneck for PRM-guided multi-agent test-time search: scoring cost that currently scales with generation cost. The O(L) reduction is architecturally clean, the theory supplies a clear justification for why single-token KV readout should not lose information relative to text, and the empirical suite covers three model scales, four datasets, three search algorithms and two topologies. Explicit strengths include complete proofs under stated assumptions (Appendix A), wall-clock latency and memory measurements on GH200, and the observation that KV scores are differentiable, opening latent-space steering that text PRMs cannot do. The same-architecture constraint and Linear Representation Hypothesis are correctly listed as limitations; the contribution remains a strong systems-plus-theory result for long-horizon MAS reasoning.","major_comments":[{"comment":"Tables 1–3 and §5.1: the central accuracy claim (“matches or strictly outperforms text-PRMs”) is established only against a same-base-model, same-LoRA-rank, same-MCTS-label Text-PRM that re-encodes the full trajectory with the adapter active on every token. Because KV transfer requires identical architecture (§8), the comparison never tests a stronger or differently sized independent text verifier. Figure 2 (left) shows smaller Text-PRMs underperform, but does not rule out a carefully tuned same-size or larger text verifier trained with different objectives or more capacity. The “outperforms” half of the strongest claim therefore rests on a single baseline family; either add at least one stronger/independent text-PRM baseline or rephrase the claim to “matches or exceeds a same-architecture text-PRM while reducing cost by orders of magnitude.”","section":null},{"comment":"§5.2 / Tables 1–2: accuracy numbers are reported without error bars, multiple seeds, or statistical tests. Several reported gains are small (0.1–1 pp) or mixed (e.g., GSM8K MCTS 0.6B: Text-PRM 57.77 vs KV-PRM 54.74). Without variance estimates it is hard to judge whether “matches or strictly outperforms” is robust, especially on AIME where n is small. At minimum, report standard errors over seeds or bootstrap intervals for the main tables.","section":null}],"minor_comments":[{"comment":"Assumption 1 and Theorem 2: the spectral decay rate α is free; a short sensitivity plot or a statement of the range of α consistent with the observed k-saturation in Figure 2 (right) would make the near-optimality claim more concrete.","section":null},{"comment":"§4.1 / Eq. (8): clarify whether the LoRA adapter is applied only to the verify-token path or also modifies the cached K/V projections; the text says “adapter disabled during generation” but the precise scope of Δθ during scoring should be explicit.","section":null},{"comment":"Figure 1 caption and cost equations: the O(d L) vs O(d L^{2}) comparison is clear, but stating the precise constants c_attn, c_ffn used for the “5000×” claim (or that it is simply L) would avoid ambiguity.","section":null},{"comment":"§6 / Table 4: KV Steering is a nice proof-of-concept; a one-sentence note on step size, number of gradient steps T, and whether the steered cache remains valid for subsequent generation would help reproducibility.","section":null},{"comment":"Appendix A.1–A.2: the two subsections are both titled “Proof of Theorem 1”; renumber for clarity.","section":null},{"comment":"Related work: a brief comparison to other cache-reuse or latent-verifier ideas (beyond compression/eviction) would better situate the novelty of using KV as the reward representation.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The systems contribution is solid and the theory is carefully scoped. The main risk for a top venue is that the accuracy claim is currently baseline-sensitive; if the authors can add one stronger text baseline or temper the wording, the paper is ready. Scope fits cs.AI / systems-for-LLM venues well."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The real news is simple: they stop re-encoding the whole trajectory for every PRM call and just attend one verify token into the generator's existing KV cache. That drops scoring from O(L^{2}) to O(L), and on their tables it does not lose accuracy—and often gains a bit—across beam, MCTS, and weighted voting on MATH/GSM8K/AIME, three Qwen3 sizes, and two MAS topologies. Latency and memory numbers (up to 37× and 34×) line up with the FLOP claim. That is a concrete, usable systems win for long multi-agent rollouts.\n\nWhat is new is treating the generation KV as the direct input representation for a process reward model, not just compressing or paging it. The capacity argument (Prop. 1) and the exponential-decay / k=1 near-optimality result (Thm. 2) under spectral decay are clean and properly proved in the appendix under the linear-representation hypothesis. Training is straightforward LoRA on MCTS Q-values; the same-data Text-PRM baseline is fair for the comparison they actually run. The KV-Steering sketch is a nice extra that text PRMs cannot do.\n\nSoft spots, in proportion: the accuracy claim is only against a same-architecture, same-LoRA, same-label Text-PRM. They do show that shrinking the Text-PRM does not close the gap, but they never pit KV-PRM against a stronger or differently sized independent text verifier. Shared architecture is an explicit limitation, so the “matches or strictly outperforms” half of the headline is conditional on that setup. No error bars, no released code, free parameters (α, LoRA rank, MCTS budget) are ordinary. None of this sinks the systems result.\n\nThis is for people building multi-agent search or long-horizon TTS who are currently paying quadratic PRM cost. The math and tables are solid enough that a serious editor should send it to referees; I would bring it to reading group and expect to cite the method when the setting matches.","headline":"Clean systems fix for the quadratic PRM bottleneck in multi-agent TTS: single-token KV readout gives real O(L) scoring with matching or better accuracy under the paper's own baseline, plus usable theory.","tokens_in":17020,"tokens_out":526,"would_cite":true,"duration_ms":5299,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Scoring multi-agent reasoning can reuse the generator's own memory instead of re-reading the text, cutting cost from quadratic to linear while matching or beating text-based scorers.","keywords":["process reward model","KV cache","test-time scaling","multi-agent systems","LoRA","beam search","MCTS","mathematical reasoning"],"falsifier":"Train matched KV-PRM and text-PRM on identical MCTS labels, then measure accuracy and wall-clock scoring cost under beam search or MCTS on MATH or AIME when trajectories exceed a few thousand tokens: if KV-PRM accuracy falls materially below text-PRM while cost savings disappear, the central claim fails.","tokens_in":17029,"feed_emoji":"⚡","tokens_out":883,"duration_ms":10320,"temperature":0.7,"pith_summary":"Process reward models guide multi-agent search by scoring intermediate steps, but today's scorers re-encode the full trajectory as text, paying a quadratic attention cost that becomes prohibitive on long rollouts. This paper shows that the key-value cache already written during generation is a richer representation of the same trajectory and can be scored with a single \"verify\" token. That changes the per-call cost from O(L^{2}) to O(L). Theory proves the cache is denser than text and that one token already captures most of the reward signal; experiments on MATH, GSM8K and AIME confirm that the resulting scorer matches or exceeds text-based PRMs under beam search, MCTS and weighted voting, while delivering thousands-fold fewer FLOPs, tens-fold lower latency and memory. The same continuous scoring also opens a path to gradient-based steering of latent messages between agents.","feed_headline":"Reuse the LLM's memory, cut reward scoring cost by 5000×","feed_subtitle":"One verify token against the generation cache matches text-based PRMs at linear cost.","key_machinery":"KV-PRM: a depth-1 readout that appends one \"verify\" token, attends it against the frozen generation KV cache through a LoRA adapter, and reads the resulting logits over \"+\"/\"−\" judgment tokens. The mechanism turns the cache from a generation byproduct into the reward model's input representation.","core_discovery":"The generation-time key-value cache is a strictly richer and more efficient input for process reward modeling than the decoded text. Conditioning a lightweight LoRA adapter on that cache with a single verify token yields scores that match or surpass full-text re-encoding PRMs while reducing scoring complexity from O(L^{2}) to O(L).","pith_inferences":["If the linear-representation premise holds more broadly, other post-generation tasks (safety checks, preference ranking, tool routing) could likewise consume the generation cache instead of re-tokenizing.","Architecture-sharing is currently a hard limit; a natural extension is a learned projector that maps one model's KV tensors into another's attention space.","The exponential decay of marginal information with readout depth suggests that multi-token verifiers will rarely justify their extra cost once a single-token adapter is well trained."],"forward_implications":["Test-time search over long multi-agent trajectories can treat process scoring as essentially free relative to generation.","Verifier cost no longer forces designers to shrink the scorer or shorten trajectories, removing a hard systems bottleneck.","Because scores are continuous functions of the cache, gradient-based optimization of latent inter-agent messages becomes possible without decoding to text.","The same k=1 readout design can be dropped into beam search, MCTS or weighted voting without changing the search algorithm itself."],"fun_headline_variants":["KV cache for process rewards: 5000× less scoring cost than text","One verify token on gen cache matches text-PRMs at O(L) cost","KV-PRM reads LLM memory, cuts reward FLOPs by 5000×","Score multi-agent rollouts from KV cache, not full re-encode","Generation KV is richer input for PRMs than decoded text"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The generator and verifier must share the same architecture so the verifier can attend the native KV tensors, and the hidden states must admit a linear semantic basis for the capacity and near-optimality proofs to hold.","fun_headline_variants_meta":{"raw":{"variants":["KV cache for process rewards: 5000× less scoring cost than text","One verify token on gen cache matches text-PRMs at O(L) cost","KV-PRM reads LLM memory, cuts reward FLOPs by 5000×","Score multi-agent rollouts from KV cache, not full re-encode","Generation KV is richer input for PRMs than decoded text"]},"model":"grok-4.5","effort":"low","cost_usd":0.00507,"raw_usage":{"total_tokens":1453,"prompt_tokens":816,"num_sources_used":0,"completion_tokens":84,"cost_in_usd_ticks":50700000,"prompt_tokens_details":{"text_tokens":816,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":553,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":816,"tokens_out":84,"duration_ms":6005,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T05:02:02.270340+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train matched KV-PRM and text-PRM on identical MCTS labels, then measure accuracy and wall-clock scoring cost under beam search or MCTS on MATH or AIME when trajectories exceed a few thousand tokens: if KV-PRM accuracy falls materially below text-PRM while cost savings disappear, the central claim fails.","supporting_citations":[],"review_version":1}