{"id":"9c08ebbe-707e-4662-82b1-82c502663807","arxiv_id":"2509.07319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A gradient-alignment influence score (GGscore) that selects the highest- and lowest-scoring old interactions for replay improves incremental neural recommendation slightly over random replay, mainly at large replay ratios.","lead":"MEGG proposes keeping the most and least influential old user-item interactions, scored by a new gradient-based metric, and replaying them when a recommender retrains on new data. The authors report consistent but small gains over random replay across four datasets and three neural recommender models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (9)'s proportionality is not derived from the defined One Step Loss Change: Eq. (7) contains a sample-dependent batch term that is dropped, and the paper's proposed fix changes the update rule, so GGscore's influence claim is unsupported.","rationale":"I read the paper as claiming two connected things: GGscore approximates One Step Loss Change, and replaying the most extreme GGscores works. The first is load-bearing because without it MEGG is an uninterpreted heuristic with no principled reason to prefer extremes. The derivation in §4.1–4.2 does not establish the proportionality: the batch-average term in Eq. (7) is sample-dependent, and the B−1→B substitution alters the update rule rather than removing a constant. This is a correctness risk internal to the paper, not a disagreement with consensus. The reader's own weakest-assumption focused on the unablated keep-both-ends rule; that is a legitimate secondary concern, and it is related because the extreme-selection rule also lacks a derivation. But the more basic issue is the metric itself. I therefore partially agree with the reader: their rationale lists Eq. (9) as concern (1), though their 'weakest assumption' singles out Eq. (10). I would keep the CONDITIONAL verdict: the empirical comparisons may be reproducible, but the paper should either repair the derivation (e.g., show the batch term is negligible or replace it with a justified surrogate) and ablate the both-ends rule, or weaken the influence claim.","tokens_in":19544,"tokens_out":8741,"duration_ms":104689,"concrete_test":"Independently re-derive and measure the rank correlation on MovieLens-1M with WDL using scheme A of §5.6: for a sample of minibatches at the epoch-4→5 transition, compute the exact ΔL_i^k using the Eq. (4) update (B−1), the GGscore V·g(z_k, θ_i), and the score after the B-denominator swap. Report Spearman correlation of exact ΔL_i^k with each score and the distribution of V·g(Z_i) relative to V·g(z_k). If exact-vs-GGscore correlation is substantially below exact-vs-swapped correlation, or if V·g(Z_i) is comparable in magnitude to V·g(z_k), the dropped batch term is not negligible and Eq. (9) fails as an approximation. If the correlation remains high, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing link in the paper is Eq. (9): it converts GGscore into an estimate of One Step Loss Change, which is what licenses the replay ranking. Re-deriving from the actual removal update in Eq. (4) gives, with V = g(D_train, θ_{i+1}), ΔL_i^k ≈ α/(B−1) [V·g(z_k, θ_i) − V·g(Z_i, θ_i)] (Eq. 7). The term V·g(Z_i) is not a constant independent of z_k: Z_i contains z_k, so discarding it changes the ranking, not just the scale. Section 4.2 first 'disregards' this term and then proposes replacing B−1 by B in Eq. (4), which eliminates the batch term but changes what is being computed—the resulting quantity is no longer the Definition 2 One Step Loss Change. The paper itself calls this a 'minor departure,' but it is the only route to Eq. (9). Thus the theoretical justification that GGscore measures influence is not established from the paper's own definitions; the empirical replay-ratio result in §5.7 (advantage only above 70%) makes the method additionally fragile in the lower-capacity regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MEGG, an experience-replay method for incremental learning in neural recommender systems. The core idea is a new per-sample influence score, GGscore (Eq. 8), defined as the dot product of a sample gradient with a reference gradient vector, which is claimed to be proportional to the One Step Loss Change (Eq. 9). MEGG retains the samples with the most extreme (lowest and highest) GGscores from a reservoir and replays them together with each incoming data block. Experiments cover three backbone models (WDL, DCN, NFM) and four datasets, with rating prediction (RMSE) and classification (AUC) metrics. The paper reports that MEGG outperforms replay baselines (iCaRL, MIR, GDumb) and incremental methods (IncCTR, SML) in most settings, approaches Full-Batch performance, and also improves IncCTR/SML when used as a replay component. Additional analyses address hyperparameters, sampling efficiency, and replay ratio.","tokens_in":19928,"tokens_out":3926,"duration_ms":47042,"significance":"If the theoretical connection were established, MEGG would be a simple, model-agnostic, and efficient replay sampler for a relatively underexplored problem: incremental learning for neural recommenders. The empirical study is a genuine strength: Table 1 reports means and standard deviations across 24 model-dataset cells, including several baselines, and multiple follow-up analyses (integration with IncCTR/SML, efficiency, hyperparameters, replay ratio) are provided. The paper also openly discloses the settings where GDumb beats MEGG and where replay ratios below 70% are not advantageous. However, the paper's central theoretical claim—that GGscore is proportional to One Step Loss Change—is not justified by the derivation as written, and the both-ends selection rule is not supported by any ablation. The contribution is therefore best viewed currently as an empirical heuristic whose theoretical interpretation and key design choice need substantial revision.","major_comments":[{"comment":"The claimed proportionality ΔL_i^k ∝ GG(z_k, θ_i | ∇θ_{i+1} L(D_train, θ_{i+1})) is not derived from Definition 2. Starting from Eq. (7), the exact first-order estimate is ΔL_i^k ≈ α/(B−1)[V·g(z_k, θ_i) − V·g(Z_i, θ_i)], where V = g(D_train, θ_{i+1}). The 'interaction term' V·g(Z_i, θ_i) is sample-dependent because Z_i contains z_k, so discarding it changes the ranking over k, not just the scale. The proposed fix—replacing B−1 with B in Eq. (4)—does cancel that term, but it changes the parameter update and therefore the quantity being computed is no longer the One Step Loss Change defined in Definition 2. The paper calls this a 'minor departure,' but it is the only route to Eq. (9). As written, the claim that GGscore measures sample influence is an unsupported construction, not a consequence of the paper's own definitions. The authors should either provide conditions under which the batc","section":"Section 4.2, Eqs. (7) and (9)"},{"comment":"The 'keep-both-ends' selection rule is not justified. If GGscore measures influence, retaining the highest-score samples is plausible, but retaining the lowest-score samples—those least aligned with the reference gradient—is not. No ablation is provided that isolates the contribution of the low-score half (e.g., min-only, max-only, middle, or random within-score-quantile variants). The only related experiment, Section 5.7, shows MEGG's advantage over GDumb appears only when the replay ratio exceeds 70%; below that, no consistent benefit is visible. Since the default configuration uses K = M − M′ = 90% of the reservoir, the reported Table 1 results do not establish that the both-ends criterion, rather than the high replay volume, is responsible for the gains. This is a load-bearing issue for the proposed algorithm's distinctiveness.","section":"Section 4.4, Eq. (10)"},{"comment":"The replay-ratio analysis directly limits the practical scope of the claims. Figure 5 shows MEGG outperforms iCaRL, MIR, and GDumb only when the replay ratio exceeds 70%, and approaches Full-Batch only above 80%. In many incremental-learning deployments, a 70–90% replay ratio of a reservoir that already holds two-thirds of all data is a large memory footprint. The abstract and introduction claim 'superior performance' without this qualification; the conclusion should be revised to state that the advantage is demonstrated at high replay ratios, and the method's behavior in lower-capacity regimes remains unexplained. This also connects to the lack of an ablation for the selection rule: the high-ratio regime is exactly where random selection also becomes competitive.","section":"Section 5.7, Fig. 5"}],"minor_comments":[{"comment":"The dataset column header 'Taobao204' appears to be a typo for 'Taobao2014'.","section":"Table 1"},{"comment":"The reference 'with (??)' is an unresolved placeholder; it should cite Eq. (10).","section":"Algorithm 1, line 4"},{"comment":"The statement that 'an AUC increase at the 0.001 level is considered statistically significant' is presented without a citation or test; as written it conflates practical significance with statistical significance. Please clarify or remove.","section":"Section 5.2"},{"comment":"The MNIST validation of One Step Loss Change is described only briefly. It is unclear how the SGD-Influence Estimator is used to compute 10 One Step Loss Change values per sample and how the cumulative Loss Change is defined. Adding details would strengthen the empirical motivation.","section":"Section 4.1 / Fig. 2"},{"comment":"The parameter selection rule (embedding parameters plus final fully connected layer) is justified only by prior work. Since the paper emphasizes efficiency, a brief statement of how many parameters are retained relative to the full model for the three backbones would be useful.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The empirical component is solid and the paper is clearly written in structure, but the theoretical centerpiece (Eq. 9) is not derived from the paper's own definitions. The both-ends selection rule is also empirically unvalidated. I believe these are fixable within a revision: the authors can either prove the proportionality under explicit assumptions, or recast GGscore and MEGG as a heuristic and add the missing ablations. I would not reject outright because the experimental evidence, especially Table 1 and the integration experiments, is valuable to the community. However, as it stands, the manuscript overstates the theoretical grounding of the method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful empirical replay-sampling paper for neural recommenders, with a real gap—how to select reservoir samples under domain-incremental drift—and a score that, despite an overclaimed derivation, is a plausible gradient-alignment heuristic. The theory in Section 4.2 is not the reason to trust it.\n\nWhat's actually new: MEGG applies the gradient-alignment idea (V·∇L) to experience replay for NCF models, restricted to user/item embeddings and the final layer for efficiency. The evaluation is broad: three backbones, four datasets, five seeds, standard deviations, and comparisons against GDumb, iCaRL, MIR, IncCTR, and SML. The paper is candid about cases where GDumb wins, and the Full-Batch parity at high replay ratios is a useful sanity check. The replay-ratio and hyperparameter analyses are more transparent than most papers in this area.\n\nSoft spots, in order of importance. The stress-test note is mostly right: Eq. (9) is the load-bearing link, and it does not follow from their own definitions. To get the clean proportionality you have to drop a batch-dependent term that contains the sample, and the paper's fix changes Eq. (4) from B−1 to B, so the quantity is no longer the One Step Loss Change defined in Definition 2. Calling that a 'minor departure' understates how much of the derivation is arranged. The GGscore may still be a sensible heuristic, but the paper has not shown that it measures sample influence—that claim rests on equations that don't close. Second, the keep-both-ends rule is never justified and never ablated. Without a min-only / max-only / middle comparison, we don't know whether the low-score half contributes anything; if it doesn't, MEGG is just a gradient-norm sampler with a fancier story. Third, the gains over GDumb are mostly small—0.1 to 0.5 percent—and several Table 1 cells are within two standard deviations. The Section 5.7 result that the advantage appears only when replay ratio exceeds 70% of a reservoir that already holds two thirds of the data further limits the practical story for bounded memory. Missing citations to TracIn and GRAD-MATCH are also real, and 'pioneering experience replay for neural recommenders' is contradicted by ADER, which is in their own reference list. No code or data is shipped, so reproducibility is currently limited.\n\nBottom line: the empirical work is real and worth referee time, but the paper needs a serious revision—fix the derivation narrative, add the ablation, soften the novelty claim. I would not cite it for the theory, and would cite it for the empirical replay comparison only if I were working in this exact area. A desk reject would be a mistake; a conditional accept after revision is the honest call.","headline":"A useful empirical replay-sampling paper for neural recommenders whose theoretical derivation in Eq. (9) does not hold as stated; it deserves a serious referee, but only a conditional accept after the derivation, the keep-both-ends rule, and the novelty claims are fixed.","tokens_in":20518,"tokens_out":3378,"would_cite":false,"duration_ms":40209,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MEGG claims a per-sample gradient score, GGscore, ranks historical user-item interactions by influence, and replaying only the most extreme-scoring samples keeps incrementally trained recommenders within about 0.1–0.2 percent RMSE of full r","keywords":["incremental learning","experience replay","recommender systems","catastrophic forgetting","sample influence","GGscore","gradient alignment","data-centric AI"],"falsifier":"Run the same three-model, four-dataset protocol with replay sets restricted to (a) only high-score samples, (b) only low-score samples, and (c) random samples of the same size; if either restricted set matches MEGG's margins over random replay, or if either half is inert, the two-sided rule is not the source of the gain.","tokens_in":19341,"feed_emoji":"🔄","tokens_out":5165,"duration_ms":60032,"temperature":0.7,"pith_summary":"MEGG argues that in incremental learning for neural recommender models, the right question is not which past samples are prototypical or near decision boundaries, but which ones actually influence training. It defines GGscore, the dot product of a sample's gradient with a reference gradient, and shows that this quantity is proportional to the One Step Loss Change, an estimate of how much removing the sample would move the loss in one optimization step. The paper's proposal is to replay the samples with the most extreme GGscores—keeping both the lowest- and highest-scoring halves of the reservoir—and it reports that this rule consistently beats random replay, prototype replay, and boundary replay across three recommender architectures and four datasets. In most settings the replay set performs within a few tenths of a percent of retraining on all historical data, which is the practical target for an online recommender.","feed_headline":"Replay the extremes, rival full retraining","feed_subtitle":"A cheap gradient score ranks past interactions; keeping the extremes keeps evolving recommenders near full-batch quality.","key_machinery":"The central object is the GGscore, a dot product between a sample's parameter gradient and a reference vector, which the paper derives as proportional to the One Step Loss Change in mini-batch gradient descent (Eq. 7–9). The reference vector is the gradient of the total loss at the converged parameters, while the sample gradient is evaluated at the penultimate-epoch parameters. The second load-bearing piece is the keep-both-ends rule (Eq. 10), which retains the lowest- and highest-scoring samples and discards the middle of the score distribution. The paper's justification for this two-sided rule is empirical rather than derived.","core_discovery":"The central claim is that the influence of a user-item interaction on a neural recommender's training can be measured by GGscore, GG(z,θ|V) = V · ∇θ L(z,θ), and that this score is proportional to the One Step Loss Change under mini-batch gradient descent. Because gradients need only be computed on the user embedding, item embedding, and final-layer parameters, the score is cheap even at large embedding sizes. Given that score, the paper claims the best replay subset is not the most prototypical or most forgotten samples but the maximally extreme ones, and it shows empirically that this keep-both-ends rule outperforms existing replay samplers and recent knowledge-distillation and meta-learnin","pith_inferences":["If the dominant signal is the highest-scoring, most gradient-aligned samples, the lowest-scoring half may be functioning as a diverse representative buffer; a min-only versus max-only ablation would separate the two mechanisms, and the paper does not run one.","GGscore's proportionality to one-step loss change suggests a standalone data-pruning or denoising tool for static training sets, a use the paper mentions as motivation but does not develop.","The same keep-both-ends rule might transfer to class-incremental or NLP continual learning, where negative-influence samples are rarely replayed; this is an untested extension.","Because the efficiency argument rests on embedding locality, the score could be computed at user or item level instead of per interaction, which the paper does not explore."],"forward_implications":["Plugging MEGG's reservoir into knowledge-distillation or meta-learning incremental frameworks gives larger gains than plugging in random replay, per the paper's combination experiments.","Because only user/item embeddings and final-layer parameters are needed for scoring, sampling cost rises only mildly as embedding size grows, unlike full-gradient samplers.","At replay ratios above 70 percent of the reservoir, MEGG dominates the other replay samplers; above 80 percent it approaches Full-Batch performance.","Since the method is model-agnostic, the same scoring and reservoir logic can be applied to any neural collaborative filtering architecture without changing its internals."],"supporting_citations":[{"why":"Supplies the influence-function notion of Loss Change that GGscore is designed to approximate.","marker":"[55]"},{"why":"Provides the SGD-influence estimator and the empirical protocol used to validate One Step Loss Change as a surrogate.","marker":"[56]"},{"why":"Contributes the single-gradient-step simplification from which the One Step Loss Change derivation starts.","marker":"[58]"},{"why":"Gives the embedding-locality evidence that justifies computing GGscore only on user/item embeddings and the final layer.","marker":"[60]"},{"why":"Shows influence can be computed on a subset of parameters, the basis of the paper's model-parameter selection.","marker":"[57]"},{"why":"Supplies the random-replay baseline that MEGG must beat and that the paper identifies as surprisingly strong.","marker":"[41]"},{"why":"Supplies the prototype-replay baseline that MEGG is compared against.","marker":"[42]"},{"why":"Supplies the boundary/interference-based replay baseline that MEGG is compared against.","marker":"[43]"},{"why":"Supplies the conceptual data module and knowledge-distillation framework that MEGG is combined with in the integration experiments.","marker":"[36]"},{"why":"Supplies the meta-learning incremental retraining method that MEGG is compared with and combined with.","marker":"[15]"}],"fun_headline_variants":["Replay the extremes, match full retraining","GGscore picks extreme samples for replay","Extreme interactions: key to incremental learning","Model-agnostic replay with GGscore","Keep the extreme user-item pairs"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The keep-both-ends rule assumes the least-aligned samples are as worth keeping as the most-aligned ones, yet the paper gives no ablation isolating the contribution of the low-score half.","fun_headline_variants_meta":{"raw":{"variants":["Replay the extremes, match full retraining","GGscore picks extreme samples for replay","Extreme interactions: key to incremental learning","Model-agnostic replay with GGscore","Keep the extreme user-item pairs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1239,"prompt_tokens":702,"completion_tokens":537,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":446,"tokens_out":537,"duration_ms":7014,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:27:04.910754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-model, four-dataset protocol with replay sets restricted to (a) only high-score samples, (b) only low-score samples, and (c) random samples of the same size; if either restricted set matches MEGG's margins over random replay, or if either half is inert, the two-sided rule is not the source of the gain.","supporting_citations":[{"cited_title":"In: International Conference on Machine Learn- ing, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the influence-function notion of Loss Change that GGscore is designed to approximate."},{"cited_title":"Advances in Neural Information Processing Systems32(2019)","cited_arxiv_id":null,"evidence_quote":"Provides the SGD-influence estimator and the empirical protocol used to validate One Step Loss Change as a surrogate."},{"cited_title":"Selective and Collaborative Influence Function for Efficient Recommendation Unlearning","cited_arxiv_id":"2304.10199","evidence_quote":"Gives the embedding-locality evidence that justifies computing GGscore only on user/item embeddings and the final layer."},{"cited_title":"In: Com- puter Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the random-replay baseline that MEGG must beat and that the paper identifies as surprisingly strong."},{"cited_title":"Online Continual Learning with Maximally Interfered Retrieval","cited_arxiv_id":"1908.04742","evidence_quote":"Supplies the boundary/interference-based replay baseline that MEGG is compared against."},{"cited_title":"A Practical Incremental Method to Train Deep CTR Models","cited_arxiv_id":"2009.02147","evidence_quote":"Supplies the conceptual data module and knowledge-distillation framework that MEGG is combined with in the integration experiments."}],"review_version":1}