{"id":"a506f581-946f-4562-b0de-a8dc1c5f3b71","arxiv_id":"2508.18756","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A redesigned memory-layer architecture with five engineering improvements reaches performance parity with 8-expert MoE at similar compute, with lower memory access and stronger long-context memorization.","lead":"UltraMemV2 is a new AI model design that keeps knowledge in large memory tables and activates only a few entries per token, and it now matches the performance of top mixture-of-experts models with far less memory traffic. It does especially well on long-context and multi-turn memory tasks, pointing to cheaper and faster large-model deployment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Flagship parity claim uses unmatched total-parameter comparison: UltraMemV2-2.5B/60B vs SeedMoE-2.5B/30B.","rationale":"The central claim is an empirical parity result, so its validity rests on whether the comparison is iso-parameter and iso-compute. The paper's own activated/total notation exposes a structural mismatch: the flagship proprietary comparison uses 2.5B/60B UltraMemV2 versus 2.5B/30B SeedMoE, a 2x difference in total sparse parameters. This is more concrete than the reader's general 'baseline quality' concern, though related. The open-source section includes matched comparisons and should be credited; those results genuinely support parity at 1B/7B. However, the paper generalizes beyond that and uses the unmatched comparison for long-context superiority. A single additional baseline run would settle the question. Since the reader's verdict is already CONDITIONAL, this concern reinforces that conditionality rather than changing it.","tokens_in":20110,"tokens_out":5836,"duration_ms":53865,"concrete_test":"Train or obtain a SeedMoE-2.5B/60B baseline trained with the exact 3.9T PT + 500B CT schedule and evaluate it on the same benchmarks as in Tables 1-2. If SeedMoE-2.5B/60B matches or exceeds UltraMemV2-2.5B/60B-top768, the parity/superiority claims collapse. If that run is infeasible, provide an iso-total-parameter comparison, e.g., UltraMemV2-2.5B/30B versus SeedMoE-2.5B/30B, and report FLOP-matched training curves.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 and Tables 1-2 present the flagship result as UltraMemV2-2.5B/60B-top768 versus SeedMoE-2.5B/30B. The notation in this paper means activated/total parameters, so this compares a 60B-total-parameter memory network against a 30B-total-parameter 8-expert MoE. The abstract's claim of 'same computation and parameters' is therefore not satisfied in the proprietary comparison that carries the headline parity and the long-context gains (+1.6, +6.2, +7.9 in Table 2). A 2x difference in total sparse parameters is a first-order confound: the result is consistent with the alternative explanation that UltraMemV2 simply has more capacity to memorize. The open-source comparison in Table 3 does match total parameters (1B/7B and 227M/1.2B), giving genuine support for parity at those scales; but the paper's central claim is not limited to that scale, and the strongest proprietary results are unmatched. No SeedMoE-2.5B/60B run is reported at the 3.9T+500B schedule, so the main comparison is not iso-parameter. If 'parameters' is meant as activated parameters only, the claim is ambiguous and should be stated as such.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"UltraMemV2 proposes a redesigned memory-layer architecture intended to close the performance gap between memory-layer models and state-of-the-art 8-expert MoE models. The five main changes are: inserting memory layers into every transformer block, simplifying implicit value expansion to a single shared linear projection, replacing value embeddings with an FFN-based value processing derived from PEER (with the activation removed), a new initialization scheme, and a rebalanced memory-to-FFN computation ratio. The paper reports proprietary SeedMoE comparisons, open-source OLMoE/Memory+/UltraMem comparisons at 227M/1.2B and 1B/7B, extensive component ablations, and a scaling study up to a 2.5B-activated/120B-total model. The headline claims are performance parity with 8-expert MoE under the same computation and parameters, superior long-context memorization, multi-round memorization and in-context learning, and a design principle that activation density matters more than total sparse parameter count.","tokens_in":20518,"tokens_out":6345,"duration_ms":59955,"significance":"If the parity claim survives a properly controlled comparison, this is a significant result for sparse model design: memory layers with very low memory access are an appealing alternative to MoE, and the paper gives concrete architectural guidance (memory layer per block, shared single projector, PEER-style values, initialization, and computation split). The paper also contributes careful open-source comparisons at two matched parameter scales, detailed ablations of value processing, head count, memory placement, sharing topologies, and learning-rate schedules, and it explicitly acknowledges training-stage limitations. These are valuable even if the proprietary flagship comparison needs revision. The strongest weakness is that the flagship proprietary comparison is not iso-parameter, which directly affects the abstract's central claim.","major_comments":[{"comment":"The flagship parity claim and the long-context advantages (+1.6, +6.2, +7.9) compare UltraMemV2-2.5B/60B-top768 trained on 3.9T PT + 500B CT to SeedMoE-2.5B/30B at the same schedule. SeedMoE-2.5B/60B is only reported at 1.6T PT + 250B CT. Since the notation is activated/total parameters, the comparison is not iso-parameter (60B vs 30B total), and the abstract's 'same computation and parameters' is not satisfied. The 2x total sparse parameter gap is a first-order confound: the results are also consistent with extra memorization capacity. Please either add a SeedMoE-2.5B/60B run at 3.9T+500B, or explicitly restrict the parity claim to iso-parameter open-source comparisons and state the proprietary comparison as not iso-parameter.","section":"Section 4.1, Tables 1-2"},{"comment":"Multi-hop reasoning drops from 34.8 (SeedMoE-2.5B/30B) to 17.7 (UltraMemV2-2.5B/60B-top768). The paper says performance variations are 'attributable to architectural differences rather than parameter count disparities' with no supporting analysis. Given the comparison is already confounded by total parameter count, this 17-point regression is a load-bearing caveat that must be reconciled or at least analyzed (e.g., by task-level ablations or a SeedMoE-2.5B/60B baseline) before the 'superior long-context learning' title claim is supported.","section":"Section 4.1, Table 2"},{"comment":"The claim 'activation density has greater impact on performance than total sparse parameter count' is supported by comparing UltraMemV2-2.5B/60B-top768 to UltraMemV2-2.5B/120B-top256, but both TopM and total parameter count change between the two runs. This is not a controlled ablation for activation density. The observed differences are small and mixed across sub-benchmarks (e.g., OpenBench knowledge 80.7 vs 80.3, but OpenBench All 68.3 vs 69.1). Please provide a controlled sweep (vary TopM under fixed total parameters and fixed compute) or rephrase the conclusion as a config comparison.","section":"Section 4.1, bullet 3; Table 1"},{"comment":"The parity claim rests on 'same computation and parameters', but the paper does not report FLOP counts, training wall-clock, or inference latency/memory-access measurements, nor baseline hyperparameter budgets for SeedMoE. Without these, the efficiency parity cannot be verified; lower memory access is asserted but not measured in this work.","section":"Section 4.1, training protocol"}],"minor_comments":[{"comment":"The open-source model is called 'UltraMemV2-1M/7B' in Table 3 and Table 10, but the text and context indicate '1B/7B'. Please correct the typo.","section":"Table 3 and Table 10"},{"comment":"Section 3.6 says the two auxiliary losses are 'NOT used in UltraMemV2', but Section 4.2.7 reports ablation experiments with and without them. Clarify that they are not used in the final model, while being studied in ablations.","section":"Section 3.6 vs Section 4.2.7"},{"comment":"The paper says it 'adopt[s] PEER' but then removes the activation function, and states 'this change will be uniformly abbreviated as PEER'. Calling a modified variant 'PEER' can confuse readers; consider a distinct name or explicit notation.","section":"Section 3.4"},{"comment":"The initialization derivation relies on calibrating the top-k score mean/variance using randomly generated data whose distribution is assumed to match training-time distributions. This is a heuristic calibration step rather than a closed-form derivation; state this clearly in the main text if it is part of the claimed 'principled initialization'.","section":"Appendix A"},{"comment":"The choice of MCP=17% and the scaling rule Dk=h/2 are based on a single small-scale ablation (500M/6B) with no error bars or multiple seeds. This may be acceptable, but the lack of uncertainty makes the rule look more deterministic than the evidence supports.","section":"Section 4.2.4"},{"comment":"There are several typos and formatting issues: 'Prelimilary' (Section 3.1), 'Resent MOEs' (Section 2), 'dots.llm1[19]' reference formatting, and inconsistent use of 'top-k'/'TopM'. These do not affect the technical content.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the unmatched total-parameter comparison in the flagship proprietary result, which is likely fixable if the authors can run SeedMoE-2.5B/60B on the 3.9T+500B schedule or if they reframe the parity claim to the iso-parameter open-source comparisons. Given the authors' affiliation and the scale already reported, a major revision seems feasible rather than requiring rejection. The paper's open-source controlled comparisons and ablations are valuable and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mark, quick take on UltraMemV2. The engineering is solid and the paper is worth reading; the central parity claim is plausible but not yet proven by the flagship numbers. The five changes (memory layers every block, simplified IVE, PEER-style values, principled init, computation rebalancing) are sensible and mostly borrowed, but the synthesis works: the open-source results at 1B/7B and 227M/1.2B show UltraMemV2 matching OLMoE 8-expert MoE under matched total parameters. That is the best evidence in the paper, and it is real support for the main claim at small scale.\n\nThe soft spot is the proprietary comparison. The long-context and extended-training results that carry the headline compare UltraMemV2-2.5B/60B to SeedMoE-2.5B/30B, not to a 60B SeedMoE. The abstract says 'same computation and parameters,' but this comparison uses twice the total sparse parameters. There is a matched SeedMoE-2.5B/60B row in Table 1, and at 1.6T+250B the two are close (Open 69.1 vs 68.1), which is encouraging, but the paper does not emphasize that and the Table 2 long-context gains (+1.6, +6.2, +7.9) come from the unmatched 30B baseline. That is a first-order confound, not a nitpick: the result could just be more capacity to memorize.\n\nAlso, the activation-density claim (top768 beats top256 despite fewer total params) compares models that differ in both activation density and total parameters, so it is confounded. The paper frames it as a principle, but it is a post-hoc interpretation of a single pair of runs. No error bars, no multiple seeds, proprietary eval details are in-house. The paper is honest about limitations (slow early training, smaller CT gains, need for memory layer in every block), which I credit.\n\nVerdict: serious referee, yes. The architecture is a plausible new data point in memory-layers-vs-MoE, and the small-scale matched results give it credibility. The authors should either run the matched SeedMoE-2.5B/60B at the 3.9T+500B schedule and the long-context suite, or visibly soften the 'same parameters' claim. I'd send it to review with that demand.","headline":"The architecture is a genuine step forward for memory layers and the small-scale matched comparisons support parity, but the flagship proprietary claim compares 60B total params against 30B, so the headline result is not yet proven at scale.","tokens_in":20944,"tokens_out":2336,"would_cite":true,"duration_ms":21285,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A redesigned memory layer matches 8-expert MoE at equal compute and parameters—and wins on long-context recall.","keywords":["memory layers","mixture of experts","sparse models","long-context learning","product key memory","inference efficiency","activation density","MoE parity"],"falsifier":"Train an 8-expert MoE and an UltraMemV2 model on identical data with matched FLOPs and equal hyperparameter-tuning effort, then compare loss curves and benchmark scores at every checkpoint: if the MoE leads at equal compute throughout training, the parity claim fails. A second checkable observation: the paper claims significantly low memory access but reports no measured per-token bytes read or serving latency, so directly measuring memory traffic in a deployed setting would test that part of the claim.","tokens_in":20070,"feed_emoji":"🧠","tokens_out":6643,"duration_ms":59037,"temperature":0.7,"pith_summary":"The paper claims that a memory-layer architecture—where the model retrieves stored values from a large table instead of routing tokens through expert networks—can now match the performance of state-of-the-art 8-expert mixture-of-experts models under equal computation and parameter counts. Previous memory-layer designs, including the authors' own UltraMem, only matched 2-expert MoE. UltraMemV2 closes that gap with five changes, and goes further on memory-heavy jobs: long-context memorization, multi-round recall, and in-context learning improve over an MoE counterpart by 1.6, 6.2, and 7.9 points respectively. The paper also argues, from a scale-up to 120B total parameters, that how many values a token activates matters more than how many sparse parameters exist. If right, this makes memory layers a credible low-memory-access alternative for large sparse language models.","feed_headline":"A memory-layer model matches 8-expert MoE at 120B scale","feed_subtitle":"Memory layers read far less at inference and win on long-context recall and in-context learning.","key_machinery":"The load-bearing object is the UltraMemV2 memory layer: row and column key scores pass through a small Tucker core to form a top-m grid over a huge factorized value table, and only the top-m values are read—which is what keeps memory access far below expert routing. The five changes—memory layer in every block, a single shared value projector, PEER-style FFN values, initialization variance matched to the FFN's, and a roughly 17% memory compute share—are what make the retrieved values competitive with routed experts.","core_discovery":"UltraMemV2 is a memory-layer architecture that places a memory layer in every transformer block, retrieves values via Tucker-decomposed query-key scoring (TDQKR), and processes the activated values with a single shared projector and an FFN-style value computation taken from PEER. With principled parameter initialization and a rebalanced memory-to-FFN compute split (about 17% to memory), the authors report that 2.5B-activated-parameter models over 60B-120B total parameters match SeedMoE baselines with 8 activated experts at equal compute and parameters, after continued training. On retention-heavy benchmarks the memory layer wins: +1.6 on long-context memorizing, +6.2 on multi-round memorizin","pith_inferences":["The paper's own table shows a large MoE advantage on multi-hop reasoning (17.7 vs 34.8) even where memory layers win overall—suggesting memory layers store and retrieve associations well but chain them poorly; testing a hybrid that keeps a few routing experts in late blocks is a natural follow-up the paper leaves implicit.","Parity is demonstrated after continued training on high-quality data; at the 1.6T-token pretraining checkpoint MoE leads on math, code, and reasoning. A FLOP-matched training-curve comparison would reveal when, and whether, the crossover happens—something the paper does not report.","If activation density is the dominant lever, the next bottleneck is retrieval cost growing with sequence length; measuring the crossover where top-m memory access exceeds expert-routing cost would decide how far this architecture class can scale."],"forward_implications":["Memory-layer architectures reach performance parity with 8-expert MoE under equal compute and parameters while reading far less memory at inference, making them a practical alternative for efficient sparse models.","On memory-intensive workloads—long-context memorization, multi-round dialogue recall, and in-context learning—memory layers beat expert routing by 1.6, 6.2, and 7.9 points respectively.","Activation density beats parameter count: with matched activated parameters, a 60B table with top-768 outperforms a 120B table with top-256 on the same benchmarks.","Training simplifies: the Tucker core penalty loss, balance loss, and decaying value learning-rate schedule—all used in prior memory-layer work—can be removed without hurting final performance.","The design scales to 2.5B activated / 120B total parameters, with the rule Dk ≈ h/2 and key count growing only as the square root of hidden size keeping the memory compute share stable."],"supporting_citations":[{"why":"The prior UltraMem architecture this work extends; supplies TDQKR and Implicit Value Expansion, and defines the 2-expert-level baseline the paper must beat.","marker":"[18]"},{"why":"PEER supplies the FFN-with-one-inner-dimension value processing adopted as a key improvement over plain value embeddings.","marker":"[12]"},{"why":"Establishes that 8 activated experts is the performance-efficiency sweet spot, defining the target configuration for parity.","marker":"[25]"},{"why":"OLMoE is the open-source 8-expert MoE baseline for controlled comparison and also provides the 1T-token open training data.","marker":"[29]"},{"why":"Product Key Memory originates the memory-layer idea and product-quantized retrieval that TDQKR builds on.","marker":"[26]"},{"why":"Memory+ serves as a memory-layer baseline in the open-source comparison and motivates the shared-memory ablations.","marker":"[2]"},{"why":"Supplies the initialization standard deviation convention (sqrt(2/5h)) adopted in the paper's variance derivation.","marker":"[30]"}],"fun_headline_variants":["Memory layers match 8-expert MoE at 120B, win on recall","UltraMemV2: 120B memory model equals 8-expert MoE","120B memory-layer model wins on recall vs 8-expert MoE","Memory layers: MoE parity, better recall, 120B scale","UltraMemV2 memory networks: 120B, matches MoE, boosts recall"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The parity claim assumes the proprietary SeedMoE baselines were trained with comparable data quality, compute budget, and hyperparameter tuning; the paper reports no FLOP-matched training curves, latency numbers, or baseline tuning details, so a weaker baseline would make the parity result an artifact of the comparison.","fun_headline_variants_meta":{"raw":{"variants":["Memory layers match 8-expert MoE at 120B, win on recall","UltraMemV2: 120B memory model equals 8-expert MoE","120B memory-layer model wins on recall vs 8-expert MoE","Memory layers: MoE parity, better recall, 120B scale","UltraMemV2 memory networks: 120B, matches MoE, boosts recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000978,"raw_usage":{"total_tokens":4027,"prompt_tokens":820,"completion_tokens":3207,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":3109}},"tokens_in":564,"tokens_out":3207,"duration_ms":23891,"temperature":1.0,"reasoning_tokens":3109,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:13:14.383662+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an 8-expert MoE and an UltraMemV2 model on identical data with matched FLOPs and equal hyperparameter-tuning effort, then compare loss curves and benchmark scores at every checkpoint: if the MoE leads at equal compute throughout training, the parity claim fails. A second checkable observation: the paper claims significantly low memory access but reports no measured per-token bytes read or serving latency, so directly measuring memory traffic in a deployed setting would test that part of the claim.","supporting_citations":[{"cited_title":"Large memory layers with product keys.Advances in Neural Information Processing Systems, 32, 2019","cited_arxiv_id":null,"evidence_quote":"Product Key Memory originates the memory-layer idea and product-quantized retrieval that TDQKR builds on."}],"review_version":1}