{"id":"1eff8834-2c96-4a7d-bdea-f969c66d3392","arxiv_id":"2502.05558","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LMN stores compressed user behavior in a shared product-quantized memory table and reports click-through and order gains on Douyin e-commerce search.","lead":"This paper describes LMN, a memory block that compresses user click history into a shared lookup table that a recommender system can search quickly. The authors report that LMN raised orders per user by 0.87 percent in a live test on Douyin's e-commerce search and say it now serves millions of users daily.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The memory-injection loss (Eq. 11) may collapse the memory output into a per-item query reconstruction; absent an ablation isolating Eq. 11 from the added query-MLP capacity, the claimed spatial/temporal memory mechanism is unsubstantiated.","rationale":"The reader's weakest_assumption correctly identifies the unvalidated memory-injection mechanism. My analysis sharpens it: the direction of Eq. 11 is not merely a missing ablation; the loss actively encourages M_O to approximate a per-item query, which would prevent the memory block from storing cross-item sequence information. Under that failure mode, the memory becomes a large learned codebook that the top-K lookup retrieves from, but because each lookup is pulled toward its own query, the aggregate sequence representation is essentially a bag of query reconstructions. The reported gains over MIMN could then be explained by the additional parameters in the user-aware query MLP and the auxiliary loss, not by the novel memory mechanism. This is a correctness risk because the paper's abstract and conclusion make mechanism-level claims (spatial perception, temporal memorization) that are not directly demonstrated. The paper does provide real deployment evidence and a scaling trend, which supports the engineering story, but the scientific claim about what the memory stores remains unverified. The proposed ablation directly tests whether removing the injection loss or freezing the values changes performance; the cosine-similarity diagnostic tests whether the memory output degenerates to the query. If the variants perform equally, the central mechanism claim would need to be softened to 'a parameter-efficient large codebook with auxiliary regularization.' Given the evidence gap, the existing CONDITIONAL verdict remains appropriate, and I recommend no change.","tokens_in":151,"tokens_out":3408,"duration_ms":43253,"concrete_test":"Run the offline comparison with three additional variants: (a) full LMN with α=0 in Eq. 12 (memory loss disabled) while keeping the query MLP, keys, and values; (b) full LMN with memory values frozen at random initialization (keys and query MLP trained); (c) full LMN with α=0 and frozen random values. If variants (a) or (b) match the reported LMN AUC of 0.7371 within 0.05%, the memory-injection mechanism is not load-bearing. Additionally, report the average cosine similarity between M_O and M_Q during training; if it approaches 1, the memory output has collapsed to the query and cannot be storing sequence-level history.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LMN compresses user history into a shared product-quantized memory block and that this memory, rather than merely added parameters, drives the reported gains. The only mechanism that writes information into the memory values is the Smooth-L1 loss in Eq. 11, which minimizes the difference between the memory output M_O (a softmax-weighted combination of top-K memory values) and the user-aware query M_Q = MLP(merge(x,u)). Because M_Q is computed from the current item x and user features u only, this loss teaches M_O to reconstruct the query for that item. It does not directly encourage M_O to contain information from other positions in the user's sequence. Over a sequence, each item's M_O is pulled toward its own query, so the sequence representation becomes a pooling of item-wise query reconstructions. The memory values may then merely store a codebook of query-like features, and the observed AUC gains could come from the extra query MLP parameters and the auxiliary loss's regularizing effect. No ablation in Section 3 separates the contribution of Eq. 11 and the learned memory values from the added capacity; the scaling experiment in Table 2 only varies the number of keys, which also scales parameters. Unless memory values are shown to carry information beyond the current query, the mechanism-level claims of spatial perception and temporal memorization are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Large Memory Network (LMN), a CTR-prediction module that compresses user behavioral sequences into a global, product-quantized memory table of keys and values. During training, top-K memory slots are retrieved with user-aware queries and a Smooth-L1 auxiliary loss injects user-item information into the memory values. The authors report offline gains on a four-week Douyin E-Commerce Search dataset (AUC +4.68% over MIMN) and an online A/B test showing +0.87% order/user and +0.72% order/search improvements, together with a GPU-sharded memory parameter server deployment. The central claim is that the shared memory mechanism, rather than added model capacity, drives these gains and enables spatial perception and temporal memorization.","tokens_in":7631,"tokens_out":3708,"duration_ms":40847,"significance":"If the central claim holds, LMN would be a practically valuable way to increase model capacity in industrial recommenders without proportional compute cost, and the described deployment architecture is a real engineering contribution. The paper also provides a scaling experiment and an online A/B test, which are useful de-risking signals. However, the manuscript does not currently isolate the memory mechanism from added parameters and auxiliary regularization, and the reported significance levels are not backed by confidence intervals or variance information. The central mechanism-level claim is therefore not yet supported by the evidence presented.","major_comments":[{"comment":"The only mechanism that writes information into the memory values is the Smooth-L1 loss between M_O and M_Q, where M_Q is computed from the current item x and user features u. This loss does not directly force M_O to contain information from other positions in the user's sequence; it can be satisfied by a codebook of per-item query reconstructions. Because Table 1 includes no ablation that disables the injection loss (e.g., alpha=0) or that replaces learned memory values with random/fixed values while holding the query MLP capacity constant, the reported AUC/LogLoss gains cannot be attributed specifically to spatial or temporal memory content rather than to the additional query MLP parameters and the auxiliary loss's regularizing effect.","section":"2.2.3, Eq. (11); Section 3.2"},{"comment":"The scaling experiment varies sqrt(n) from 50 to 500, which simultaneously increases the number of memory keys and memory values and therefore the parameter count. Without controlling for total parameters or measuring memory-slot utilization, the monotone improvement in AUC is equally consistent with a conventional capacity-scaling effect and does not substantiate a memory-specific scaling law. In addition, Section 3.1 states that the number of memory keys is set to 300 while Table 2 labels rows by sqrt(n), leaving the actual value of n and the relation between n and sqrt(n) ambiguous; please define both and report the resulting parameter counts for each row.","section":"Table 2; Section 3.1"},{"comment":"The online A/B test reports p=0 for the two commercial metrics, and Table 1 reports p<0.05, but neither result is accompanied by confidence intervals, variance estimates, the number of experimental units analyzed, or the exact significance test used. With more than 160 million users in the online test, even a negligible effect can be statistically significant, and a p-value of exactly 0 is not a meaningful report for a continuous metric. Please provide effect sizes with confidence intervals and describe the statistical procedure; otherwise the claim of 'substantial significance' is not verifiable.","section":"3.3, Table 3; Table 1"}],"minor_comments":[{"comment":"The 'cross operation' in Eq. (3) and the broadcast operation in Eq. (9) are not formally defined; please specify exactly how M_K is constructed from the row and column keys and how S_row and S_col are combined into the two-dimensional score matrix.","section":"Eq. (3) and Eq. (9)"},{"comment":"The memory loss weight alpha is never given a value; please report the chosen value and, ideally, a small sensitivity analysis.","section":"2.2.4, Eq. (12)"},{"comment":"The online latency cost is reported only as a 0.38% increment; please include the absolute serving latency before and after deployment and the measurement methodology.","section":"3.3"},{"comment":"The dataset and code are proprietary, and preprocessing and hyperparameter details are minimal; adding more details (sequence length used, embedding initialization, training configuration) would improve reproducibility.","section":"3.1"}],"recommendation":"major_revision","confidential_remarks":"The skeptic's concern about Eq. (11) is well grounded: the memory-injection loss is the only mechanism that writes into the values, and no ablation isolates it from added capacity. The evaluation metrics themselves are independent of the auxiliary loss, so I do not see a circularity problem. Given the industrial setting, proprietary data and code are not disqualifying, but the mechanism attribution must be fixed before the paper can support its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: LMN is a legitimate engineering adaptation of product-key memory to CTR prediction, and the online A/B gains are worth taking seriously. But the paper's own mechanism claims—spatial perception and temporal memorization—are not actually supported, because the memory-injection loss in Eq. 11 may just be teaching the memory to reconstruct the current query per item. Without an ablation that isolates the memory values from the added query-MLP capacity, the offline gains could come from extra parameters or a regularizing auxiliary loss. That said, the paper is honest, properly cites its building blocks, and reports a real deployment. It is the kind of contribution that deserves a serious referee, not a desk reject.\n\nWhat is new: the combination of product-quantized shared memory with user-aware queries and Smooth-L1 memory injection for CTR prediction is not in the cited prior work. The deployment framework (MPS) is practical and is described concretely. The offline comparison on 3.6B samples and the online A/B on 160M users are substantive, and the reported gains, while small in absolute terms, are typical for CTR.\n\nSoft spots: the scaling experiment in Table 2 varies the number of memory keys, which also scales parameters, so it doesn't isolate memory capacity. The significance reporting (\"p=0\", \"p<0.05\") lacks confidence intervals or variance estimates; for an A/B test with 160M users, p=0 is plausible but the reader can't check. Code and data are proprietary, which is normal for an industry paper but limits verification. The self-citation [3] is not an issue.\n\nThe stress-test concern about Eq. 11 resonates. If the memory output is pulled toward the query for each item, then the memory values may just become a codebook of query-like features, not a store of sequence-level history. The paper would be considerably stronger with an ablation that removes Eq. 11 or replaces the memory values with random fixed values while keeping the query MLP.\n\nBottom line: this is a solid industrial contribution with a weak mechanism story. A good referee would ask for that ablation. I would not cite it as evidence for memory mechanisms, but I would cite it as a deployed instance of product-key memory in recommender systems. Bring it to a reading group if you care about practical memory networks.","headline":"An honest industry adaptation of product-key memory with real online gains, but the mechanism claims need an ablation that the paper doesn't provide.","tokens_in":8214,"tokens_out":3005,"would_cite":true,"duration_ms":27813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that routing user behavior through a large shared, product-quantized memory block improves recommendation accuracy and can be deployed at million scale with negligible serving cost, with absolute gains shown on Douyin…","keywords":["large memory network","sequential recommendation","click-through rate prediction","product quantization","user behavior modeling","memory parameter server","online deployment"],"falsifier":"Retrain LMN on the same dataset with the memory-injection term removed ($\\alpha=0$) and with the memory values frozen at random initialization, keeping the query MLP and key tables trainable. If AUC and LogLoss remain close to the reported 0.7371 and 0.5986, the memory-storage claim is not what drives the gains; if performance drops back toward the MIMN baseline, the injection is doing the work.","tokens_in":7151,"feed_emoji":"🛒","tokens_out":6405,"duration_ms":55643,"temperature":0.7,"pith_summary":"The paper tries to establish that a recommender can compress each user's behavior history into a large memory table shared by all users, and that reading from this table with a learned query improves click-through prediction. If true, the model gains a form of cross-user spatial generalization and long-term temporal memory without paying the full cost of processing long sequences. The authors report that LMN beats the previous memory-based method MIMN by 4.68% in AUC and 1.46% in LogLoss offline, and raises order/user by 0.87% and order/search by 0.72% in an online A/B test. They also report full deployment at Douyin E-Commerce Search with only 0.38% added serving latency.","feed_headline":"Memory block lifts e-commerce search orders per user by 0.87%","feed_subtitle":"A shared product-quantized memory table stores user histories for long-term recall with little added serving latency.","key_machinery":"The load-bearing object is the product-quantized memory block: a shared key matrix $M_K \\in \\mathbb{R}^{n\\times d}$ decomposed as $M_K = M_{K-\\text{row}} \\times M_{K-\\text{col}}$ with both factors in $\\mathbb{R}^{\\sqrt{n}\\times d}$, plus a value matrix $M_V$ holding the stored information. The row/column decomposition reduces activation cost from $O(nd)$ to $O(\\sqrt{n}d)$; a user-aware query MLP merges the item embedding with user features, top-K scores are selected by softmax, and values are combined by weighted pooling. Memory injection is carried by a Smooth-L1 loss $\\text{Loss}_{\\text{Memory}} = \\text{SmoothL1}(M_O, M_Q)$ that pushes the memory output toward the query, trained jointly with the CTR loss.","core_discovery":"The central claim is that storing compressed user-history information in a global, large-scale memory block, rather than in per-user model weights, lets the recommender both perceive shared interests across similar users and retain long-term interests that fixed-length sequences forget. LMN extracts top-K memory values by a product-quantized two-axis lookup: memory keys are decomposed into row and column keys, so retrieving a value costs $O(\\sqrt{n})$ instead of $O(n)$. A user-aware block conditions the memory query on the user embedding, and a Smooth-L1 loss between memory output and query injects user-item interaction information into the shared memory values during training. The paper argues that the resulting memory mechanism is lightweight enough to scale to million-slot tables in production, and that its global sharing is what explains the observed offline and online gains.","pith_inferences":["Not stated in the paper: an ablation with the memory-injection coefficient $\\alpha=0$ and frozen memory values would separate genuine memorization from the added capacity of the query MLP and auxiliary loss.","Not stated in the paper: the temporal-memory claim implies that users with longer histories should benefit most; a subgroup analysis by history length would test this directly.","Not stated in the paper: the same shared memory could memorize outputs of other modules, such as feature-cross or compressor MLPs, which the paper's conclusion mentions only as future direction."],"forward_implications":["Larger memory tables are directly beneficial: varying the per-axis key count from 50 to 500 improves AUC from 0.7357 to 0.7380 and LogLoss from 0.5997 to 0.5978.","LMN is a plug-and-play module: it can be attached to an existing CTR model and trained end-to-end with $\\text{Loss}_{\\text{LMN}} = \\text{Loss}_{\\text{CTR}} + \\alpha \\text{Loss}_{\\text{Memory}}$.","Because retrieval cost grows only as $\\sqrt{n}$, the memory can be scaled to million-slot capacity with limited compute growth.","A global shared table makes different users' behavior mutually influence retrieval, providing the spatial generalization that per-user sequence models lack."],"supporting_citations":[{"why":"supplies the product-key memory decomposition that lets LMN reduce lookup cost from O(n) to O(sqrt n)","marker":"[9]"},{"why":"defines the MIMN memory-network baseline that LMN improves on in AUC and LogLoss","marker":"[12]"},{"why":"provides product quantization, the basis for decomposing memory keys into row and column factors","marker":"[8]"},{"why":"gives the key-value memory network abstraction that LMN's query-key-value retrieval builds on","marker":"[11]"},{"why":"supplies the Smooth-L1 loss used for memory injection in Eq. (11)","marker":"[6]"},{"why":"is the DIN target-attention baseline compared in the offline experiments","marker":"[19]"},{"why":"is the SIM long-sequence search-based baseline compared in the offline experiments","marker":"[13]"}],"fun_headline_variants":["Shared memory block lifts e-commerce orders by 0.87%","Million-scale memory table boosts recommendation orders 0.87%","Large memory network: +0.87% orders in live A/B test","Compressed user history in global block yields 0.87% order lift","Forget-me-not: shared memory block raises orders 0.87%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that the Smooth-L1 loss in Eq. (11) genuinely writes useful user-item information into the shared memory values; if that loss only acts as regularization, the reported gains could come from the extra query-MLP parameters and auxiliary loss rather than from the memory mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Shared memory block lifts e-commerce orders by 0.87%","Million-scale memory table boosts recommendation orders 0.87%","Large memory network: +0.87% orders in live A/B test","Compressed user history in global block yields 0.87% order lift","Forget-me-not: shared memory block raises orders 0.87%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3300,"prompt_tokens":873,"completion_tokens":2427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2330}},"tokens_in":489,"tokens_out":2427,"duration_ms":16910,"temperature":1.0,"reasoning_tokens":2330,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:49:57.802387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain LMN on the same dataset with the memory-injection term removed ($\\alpha=0$) and with the memory values frozen at random initialization, keeping the query MLP and key tables trainable. If AUC and LogLoss remain close to the reported 0.7371 and 0.5986, the memory-storage claim is not what drives the gains; if performance drops back toward the MIMN baseline, the injection is doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the product-key memory decomposition that lets LMN reduce lookup cost from O(n) to O(sqrt n)"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"gives the key-value memory network abstraction that LMN's query-key-value retrieval builds on"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Smooth-L1 loss used for memory injection in Eq. (11)"}],"review_version":1}