{"id":"f593eba5-5668-4b78-8b7d-7467741f77fc","arxiv_id":"2411.09425","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MARM caches per-item attention results to replace O(n^2) masked self-attention with O(n) target-attention, enabling deeper sequence modeling at low FLOPs.","lead":"Paper: MARM is a recommendation model that caches intermediate attention outputs, letting it run several stacked attention layers at nearly the cost of one layer. It reports 0.43% offline GAUC gains and 2.079% online watch-time gains per user on a deployed short-video platform.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central O(n^2·d)→O(n·d) equivalence is unvalidated: MARM's cached K/Vs are frozen and gradient-detached (Sec 4.0.2), never compared to the exact uncached multi-layer attention model, so the 'cache scaling law' may confound memory with FLOPs/parameters.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: MARM's cached representations are frozen and outside the computation graph, so the trained model is not the multi-layer attention model whose complexity is claimed to be reduced. The paper's own Sec. 4.0.2 acknowledges this and defends it with a heuristic about streaming one-epoch training and slowly changing lower-layer parameters, but provides no quantitative comparison against the exact uncached architecture. The deployed A/B results and the 60TB system are genuine evidence of practical utility, so the correct verdict remains CONDITIONAL rather than REJECT: the mechanism may work in production, but the central scientific claim—that caching preserves the multi-layer attention modeling while reducing complexity—is unverified. The proposed concrete test would settle whether the approximation is faithful and whether the cache content, rather than added parameters or FLOPs, drives the reported gains. Because this concern reinforces the reader's conditionality without changing it, the recommendation is UNCHANGED.","tokens_in":15475,"tokens_out":8943,"duration_ms":84022,"concrete_test":"Run a controlled offline experiment at n = 50 (or 100) with the same L and d: (1) train the exact uncached multi-layer masked-self-attention model under the same streaming one-epoch protocol; (2) train MARM with a freshly warmed cache updated each step (or refreshed at each epoch) and with the frozen-cache variant; (3) compare GAUC and loss. If the exact model does not match MARM within a small margin, the equivalence claim is false. Additionally, replace MARM's cached vectors with random frozen vectors of the same shape; if the random-cache model retains most of the improvement, the gains are not from cache content, and the 'cache scaling law' is not a memory effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.0.2 concedes that cached key/value vectors are not nodes in the computation graph and are written once and frozen, while Q/K/V and FFN parameters keep updating. The headline claim—Eqs. (5)–(6) replace an O(n^2·d) masked-self-attention stack with an O(n·d) target-attention stack—requires the cached vectors at layer l to match the true layer-l hidden states of the history items under current parameters. This is not established: the cached vectors are produced by target-attention with a candidate query rather than causal masked self-attention over the history prefix; they are stale after the first write; and no gradient reaches the layers that generated them. The paper's streaming-data argument is qualitative only, and no experiment compares MARM to the exact uncached multi-layer attention model it is designed to emulate on a tractable sequence length. Without such a comparison, the offline/online gains could be attributed to the added TA-layer parameters, to the cached vectors acting as auxiliary dense features, or to the extra FLOPs actually incurred—since C = L·n·d is proportional to the module's FLOPs, the Figure 5 'cache scaling-law' is not distinguishable from an ordinary compute/parameter scaling curve. The central memory-for-computation claim therefore lacks a direct validity check.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MARM, a memory-augmented sequence modeling module for industrial recommendation ranking. The core idea is to cache intermediate outputs of a multi-layer masked self-attention stack in an external key-value store, then approximate the stack at inference/training time by a sequence of target-attention layers that read the cached vectors, reducing per-candidate complexity from O(n^2·d) to O(n·d). The authors report a 'cache scaling law' in which performance improves with cache size C = L·n·d, offline GAUC gains of 0.43% over a strong ensemble baseline, and online A/B gains of 2.079% in play-time per user, with deployment at Kuaishou across retrieval, cascading, and ranking stages. The paper also describes a 60 TB cache storage center and resource-cost estimates.","tokens_in":15778,"tokens_out":3103,"duration_ms":29757,"significance":"If the central claim holds, the paper makes a practically significant contribution: it introduces a storage-for-compute trade-off in streaming RecSys, extends single-layer target-attention user modeling to multiple layers at controlled FLOPs, and demonstrates gains in a deployed system with online A/B validation. The scale of the deployment and the multi-stage online results are genuine strengths. However, the conceptual novelty over LLM KV-cache ideas is incremental, and the paper's core scientific assertion—that cached frozen representations faithfully emulate a full multi-layer attention model—is not directly validated. The significance therefore depends on whether that equivalence can be established experimentally, which the current manuscript does not do.","major_comments":[{"comment":"The central equivalence claim is not tested. Section 4.0.2 states that cached key/value vectors are not nodes in the computation graph, are written once, and are frozen, while Q/K/V and FFN parameters continue to update. Equations (5)-(6) replace a multi-layer masked self-attention stack with target-attention over these cached vectors, and Figure 2 claims 'The results are same!' No experiment compares MARM against the exact uncached multi-layer attention model on a tractable sequence length, nor against a version in which the cache is refreshed with current parameters. Without such a comparison, the offline and online gains could be attributed to the added target-attention parameters, to the cached vectors acting as additional dense input features, or to the extra FLOPs actually incurred, rather than to faithful approximation of deep self-attention. This is a load-bearing gap for the paper's main claim.","section":"Section 4.0.2, Eqs. (5)-(6), Figure 2"},{"comment":"The 'cache scaling law' confounds cache size with compute and parameters. The paper defines C = L·n·d and notes that 'the size of C is linearly positively correlated with the online latency' and with all training, inference, and storage resources. Figure 5(c) plots performance against sequence length for curves labeled by C, where equal C corresponds to equal FLOPs (e.g., 400*128 has variants 4*100*128, 2*200*128, 1*400*128). Thus the observed improvement could be an ordinary scaling curve in FLOPs, depth, or sequence length, not a specific effect of cache size. The paper needs an ablation that holds FLOPs and parameter count fixed while varying cache freshness, cache coverage, or cache staleness, or a comparison against an uncached model with the same total FLOPs, to support the claim that memory, rather than compute, is the scaling variable.","section":"Section 3.2.3, Figure 5(c)"},{"comment":"The comparison with HSTU* is not matched and therefore does not support the conclusion that MARM is preferable to a direct multi-layer self-attention approach at equivalent cost. Table 2 reports HSTU* with only item ID, author ID, tag, and user feedback features, a history length of 2000, depth 4, and 1.04B FLOPs, while MARM uses richer features, the existing TWIN module as its first layer, length 6000, depth 4, and 302.6M FLOPs. The two systems differ simultaneously in architecture, input features, sequence length, and training procedure, so the result that 'directly adding an HSTU-style module provided no significant improvements' is not attributable to MARM's caching mechanism. A matched comparison at a tractable scale is needed.","section":"Section 3.3.3, Table 2"}],"minor_comments":[{"comment":"The text says 'maximum attention depth L of 41'; this appears to be a typo for L = 4, since the experiments use L = 1, 2, and 4.","section":"Section 3.3.1, MARM bullet"},{"comment":"The heading 'Complexity and Resoruces' contains a typo; it should read 'Complexity and Resources'.","section":"Section 4.0.1"},{"comment":"Equation (4) uses the same notation on both sides of the equality: the hash keys [Iid1_i_UidX, ...] are passed to MARMCacheLookUp, and the returned cached values are also written as [Iid1_i_UidX, ...]. The notation should distinguish keys from values to avoid ambiguity.","section":"Section 2.1.2, Eq. (4)"},{"comment":"The phrase 'the the bottom target-attention input' contains a duplicated article and should be corrected.","section":"Section 2.1.2"},{"comment":"The label 'C=0*128' is unclear because C is defined as L·n·d; since n and d are positive, C=0 only if L=0. The figure caption should clarify what 'C=0*128' represents (e.g., the L=0 baseline).","section":"Figure 5(c)"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a real deployment with meaningful online gains, which is a strong practical result. My recommendation is driven by the gap between the paper's conceptual claim—that cache-based target attention exactly or closely emulates multi-layer self-attention—and the lack of any direct validation of that claim. The authors should add a controlled comparison on a tractable setting, and should also test the cache-size scaling law while controlling for FLOPs and parameters. If those experiments cannot be done within the current deployment, the paper should be reframed as an empirical industrial system paper rather than as establishing a new scaling law."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is simple and worth taking seriously: MARM caches layer-wise target-attention outputs for user-item pairs, so a multi-layer attention stack over user behavior can be served as a sequence of target-attention layers with cache lookups. That turns O(n^2*d) masked self-attention into O(n*d) at inference. The idea is adapted from LLM KV caches, but the application to RecSys interest modules, with training-time cache accumulation and frozen cached states, is not in the cited prior work. The paper also ships real evidence: a 60TB cache, 100 A10 GPUs, online A/B results across retrieval, cascading, and ranking, and a deployed system serving tens of millions of users. That counts for a lot. The FLOPs tables and the online watch-time gains are the strongest parts.\n\nThe soft spots are real but not fatal. The central claim that cached representations are equivalent to running the full multi-layer self-attention model is asserted (Figure 2 says 'The results are same!') but never checked. Section 4.0.2 is honest about the differences, frozen, gradient-detached K/Vs, updated only once, but the defense is a qualitative streaming-data argument. No experiment compares MARM against the exact uncached multi-layer attention model, even on a small sequence length where that comparison would be tractable. Without that, the gains could partly come from the extra target-attention parameters or from treating cached vectors as dense features, not from successfully emulating deeper attention. Relatedly, the 'cache scaling law' is not a law; it is a monotone trend, and since cache size C = L*n*d is proportional to the module's FLOPs, the curves in Figure 5 are not clean evidence that memory, rather than added compute or parameters, drives the improvement.\n\nThe paper's framing oversells. Calling this a 'milestone' and a 'scaling-law' does the actual engineering a disservice. But the engineering itself is plausible and the deployment evidence is substantial. The citation pattern is fine; using DIN, SIM, and TWIN as building blocks is natural, and the KV-cache lineage is acknowledged. The paper deserves a serious referee. I would send it to peer review and ask for two things: a small-scale comparison to the exact uncached multi-layer model, and language that stops calling a monotone curve a scaling law. Then it's a solid industrial paper.","headline":"A genuinely useful industrial caching trick for user-sequence modeling, wrapped in overclaimed 'scaling-law' language and missing the one comparison that would validate the core equivalence.","tokens_in":707,"tokens_out":910,"would_cite":true,"duration_ms":29236,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By caching intermediate masked-self-attention outputs, MARM makes multi-layer user-interest modeling affordable at linear per-candidate cost and reports a cache scaling law.","keywords":["recommendation systems","cache scaling-law","user interest modeling","target attention","masked self-attention","model FLOPs","streaming training","memory augmentation"],"falsifier":"Train the exact uncached multi-layer masked self-attention model with the same depth, sequence length, features, and streaming data, and compare its GAUC to MARM; if the exact model beats MARM by more than the reported margins, the frozen-cache approximation is the reason. A cheaper probe is to periodically recompute cached entries with current parameters and check whether GAUC rises.","tokens_in":15275,"feed_emoji":"📈","tokens_out":11265,"duration_ms":95798,"temperature":0.7,"pith_summary":"Industrial recommendation is compute-bound: models serve billions of daily samples within milliseconds, so adding parameters or data is cheap while adding operations is expensive. This paper proposes MARM (Memory Augmented Recommendation Model), which attacks that bottleneck by caching the intermediate outputs of masked self-attention layers over a user's item history. At serving time, a candidate item only pays for a target-attention pass — attention from the candidate query to the stored vectors — so per-candidate cost drops from $O(n^2 d)$ to $O(n d)$ and a single-layer interest module can become multi-layer. The paper reports a cache scaling law: quality improves as cache size $C = L \\cdot n \\cdot d$ grows, with production A/B tests showing 0.43% GAUC offline and 2.079% more playtime per user online.","feed_headline":"Caching turns recommender attention cost from quadratic to linear","feed_subtitle":"MARM stores attention outputs to stack user-history layers, lifting online watch time by 2.079 percent.","key_machinery":"The load-bearing object is the external cache memory: a key-value store whose keys identify a (user, historical item, layer) triple and whose values are that layer's masked self-attention output vectors. In the MARM forward pass, the first layer is ordinary target attention from the candidate item's embedding to the item sequence; each later layer is another target attention from the previous layer's query output to the cached vectors of the corresponding layer. This replaces the quadratic full-sequence masked self-attention with a chain of linear target attentions, and it is what makes cache size $C = L \\cdot n \\cdot d$ the resource that controls both storage and performance. Because cached values are written once and frozen, the backpropagation graph connects only the query path, a deliberate approximation to the true multi-layer model.","core_discovery":"The central claim is that caching lets a recommendation model deepen its user-interest module without paying the full cost of multi-layer attention. MARM stores, for every user, historical item, and attention layer, the masked self-attention output vector of that item; a candidate item's query then attends over these stored vectors layer by layer through target attention. Because the expensive self-attention computation is done once and reused across all candidate items, the per-candidate complexity falls from $O(n^2 d)$ to $O(n d)$, where $n$ is the sequence length and $d$ is the representation dimension. The paper further claims a cache scaling law: model quality rises with cache size $C = L \\cdot n \\cdot d$, and configurations with the same cache size behave comparably once $C$ is large enough. In the production short-video setting, the deployed MARM (depth $L=4$, sequence length $n=6000$, $d=128$, 60TB cache) reports 0.43% GAUC improvement offline, 2.079% playtime per user online, and one-eighth the combined compute and storage overhead of an uncached multi-layer self-attention module.","pith_inferences":["Editorial inference: the equal-performance-at-equal-cache-size pattern points to a compute-storage equivalence in attention-based sequence modeling, but the paper only tests a limited grid, so the equivalence should not be extrapolated far beyond the measured range.","Editorial inference: because cache entries are frozen in a one-epoch streaming regime, the method is most likely to pay off where user-item exposures are not replayed; recommendation systems with re-exposure or multi-epoch training could see the frozen cache become stale.","Editorial inference: the cached vectors are per-user, per-item interaction summaries, so the same store could unify retrieval, cascading, and ranking; the paper reports using the cache in all three stages but does not ablate the contribution of each."],"forward_implications":["A ranking model can grow user-history depth to several layers (e.g., $L=4$) and sequence length into the thousands (e.g., $n=6000$) while keeping per-candidate FLOPs linear, measured at roughly one-eighth the cost of an uncached multi-layer self-attention module.","Cache size $C = L \\cdot n \\cdot d$ behaves as a scaling resource: within the tested range, increasing $C$ improves GAUC, and configurations with the same $C$ (trading depth against sequence length) reach similar performance once $C$ is large.","The cache can be shared across recommendation stages — retrieval, cascading, and ranking — so the same stored interest representations serve multiple models rather than being recomputed.","MARM can be stacked on top of existing interest modules; the largest ensemble experiment shows 0.22% AUC and 0.43% GAUC gains over a baseline that already includes DIN, SIM, TWIN, and TWIN V2.","Online A/B tests show the deployed version adds roughly 15 ms per request and delivered +2.079% average playtime per user over the multi-stage system."],"supporting_citations":[{"why":"supplies the single-layer target-attention user-interest module that MARM extends into multiple layers","marker":"[23]"},{"why":"introduces the GSU/ESU two-stage lifelong history search that MARM uses to retrieve relevant cached items per layer","marker":"[11]"},{"why":"provides the two-stage interest network that MARM reuses as its first layer and as a baseline to stack on","marker":"[4]"},{"why":"gives the ultra-long-sequence baseline that MARM outperforms while using a shorter 6000-item history","marker":"[13]"},{"why":"defines the multi-layer masked self-attention architecture that MARM approximates with caches and that serves as the uncached FLOPs comparison","marker":"[18]"},{"why":"supplies the language-model scaling-law framing that MARM replaces with a cache-size scaling law","marker":"[9]"},{"why":"defines the attention operation underlying both the target-attention layers and the cached self-attention layers","marker":"[14]"}],"fun_headline_variants":["Caching makes recommender attention linear","Memory cache cuts recommendation compute to linear","MARM: cache scaling for recommendation systems","Attention caching unlocks scalable recommendation models","Cache-driven recommendation: quadratic to linear attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that intermediate results written to the cache once and then frozen stay accurate enough as the model's other parameters keep updating, so that the cached pipeline still behaves like a truly multi-layer attention model.","fun_headline_variants_meta":{"raw":{"variants":["Caching makes recommender attention linear","Memory cache cuts recommendation compute to linear","MARM: cache scaling for recommendation systems","Attention caching unlocks scalable recommendation models","Cache-driven recommendation: quadratic to linear attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000178,"raw_usage":{"total_tokens":1305,"prompt_tokens":959,"completion_tokens":346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":575,"tokens_out":346,"duration_ms":4021,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:38:36.315936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the exact uncached multi-layer masked self-attention model with the same depth, sequence length, features, and streaming data, and compare its GAUC to MARM; if the exact model beats MARM by more than the reported margins, the frozen-cache approximation is the reason. A cheaper probe is to periodically recompute cached entries with current parameters and check whether GAUC rises.","supporting_citations":[],"review_version":1}