{"id":"ac7142ea-530f-4964-9a8f-34e77eae974c","arxiv_id":"2505.02844","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FeSAIL combines staleness-weighted maximum coverage sampling with a staleness-scaled embedding-update regularizer, improving incremental CTR prediction by 0.3% to 1.7% AUC over baselines on four datasets.","lead":"A new training method for click-through-rate prediction models replays old data containing 'stale' features to keep embeddings fresh, improving accuracy when models are updated incrementally. The technique could help large recommender systems adapt to shifting feature distributions without full retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inverse-staleness priority in Eq. (2) is asserted but never validated; if reappearance probability is not decreasing in staleness, SAS may preserve the wrong samples and the claimed AUC gains could be an artifact of the coverage objective rather than staleness awareness.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the inverse-staleness priority rule in Section 3.2 is assumed, not tested. I agree that this is the most load-bearing point because it connects the method's core innovation to the empirical claim. The paper is otherwise coherent: the maximum coverage formulation is standard, the greedy approximation argument is sound, the neighbor-based efficiency trick is plausible, and the ablation study shows that removing either SAS or SAR hurts performance. Those are genuine strengths. There are other concerns, such as missing error bars, an unreported lambda value, and unclear baseline tuning, but those affect the strength of the empirical evidence rather than the validity of the mechanism. The inverse-staleness priority, by contrast, is the conceptual foundation for why SAS should work at all. A direct empirical check of reappearance probability, or the reversed-weight ablation proposed here, would settle whether the priority rule actually contributes to the observed gains. Since the reader already rated the paper CONDITIONAL and this concern is the same one, the verdict should remain unchanged: CONDITIONAL, pending validation of the staleness-reappearance assumption.","tokens_in":12242,"tokens_out":8669,"duration_ms":87749,"concrete_test":"Run FeSAIL with the weight in Eq. (2) reversed (w_i = s_i + b, so larger staleness gets higher priority) on Criteo, iPinYou, and Avazu, holding the reservoir size, SAR, and all training hyperparameters fixed; compare average AUC over the ten incremental periods. If reversed-weight FeSAIL matches or beats the proposed inverse-weight FeSAIL, the inverse-staleness priority is not the source of the gains and the motivation in Section 3.2 is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 justifies the SAS weights in Eq. (2) by asserting that \"features with smaller staleness will be covered with higher priorities because they might have higher possibilities of reappearing again in future datasets.\" This assertion is load-bearing: it determines the monotone inverse weighting (1/s_i + b) used in the maximum weighted coverage objective in Definition 1, and therefore controls which stale samples survive in the fixed-size reservoir. The reported 1.21% average AUC improvement over ASMG depends on SAS selecting the right historical samples, not merely any high-coverage subset. The paper never measures the empirical relationship between staleness and future reappearance. Figure 6 only reports feature drop ratios by staleness group, which confirms that the algorithm implements the priority rule, not that the priority rule is correct. If reappearance probability is flat or increasing in staleness, the inverse weighting preserves the wrong samples, and the improvement over unweighted coverage or RSS could shrink or reverse; the central empirical claim would then rest on the generic coverage objective rather than on staleness awareness. This is an addressable gap, but it is the weakest link in the causal story from observed staleness degradation to the FeSAIL design choices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper identifies a 'feature staleness problem' in incrementally trained CTR prediction models: when a feature disappears from recent incremental data, its embedding is not updated and becomes incompatible with the higher layers, degrading AUC when the feature reappears. The authors propose FeSAIL, which combines a staleness-aware sampling algorithm (SAS) that selects a fixed-size reservoir of historical samples by solving a maximum weighted coverage problem with a greedy (1-1/e) approximation guarantee, and a staleness-aware regularization mechanism (SAR) that penalizes embedding updates according to feature staleness. FeSAIL is instantiated on an Embedding&MLP model and evaluated on Criteo, iPinYou, Avazu, and a private Media dataset against six baselines, with ablations, parameter sensitivity, a case study, and runtime measurements. The paper claims an average 1.21% AUC improvement over state-of-the-art baselines.","tokens_in":12526,"tokens_out":9939,"duration_ms":93747,"significance":"The problem addressed is real and practically relevant for production recommender systems that retrain incrementally. The formulation of stale-sample replay as a maximum weighted coverage problem with a standard approximation guarantee is clean, and the proposal of a model-agnostic plug-in is attractive. The paper ships code, covers four datasets including a private industrial one, and includes ablations and sensitivity analyses. If the performance claims were solidly supported, FeSAIL would be a useful contribution to the incremental CTR training literature. The main reservations are an internal inconsistency in the reported improvement numbers, an underspecified SAR mechanism with a missing key hyperparameter, an unvalidated load-bearing assumption about inverse-staleness weighting, and insufficient statistical reporting for the small observed gains.","major_comments":[{"comment":"The RI values in Table 2 are internally inconsistent with the reported AUC values. For example, on Criteo, FeSAIL AUC 0.7553 versus ASMG 0.7502 gives a relative improvement of (0.7553-0.7502)/0.7502 ≈ 0.68%, not the listed 0.95%; on iPinYou the calculation gives ≈1.24% versus the listed 1.38%; on Avazu it gives ≈0.35% versus 0.78%; and on Media it gives ≈1.10% versus 1.72%. Since the headline '1.21% average AUC improvement' is simply the mean of the four listed ASMG RI values (0.95, 1.38, 0.78, 1.72), the central empirical claim is not currently supported by the table. The authors should recompute the RI values, or report per-time-span improvements with the underlying data, and revise the abstract and conclusion accordingly.","section":"Section 4.2, Table 2; Abstract and Conclusion"},{"comment":"The SAR mechanism is not fully specified. In Eq. (3), the guard term g_i is defined using Δe_i, described as 'the embedding change for feature f_i between two consecutive mini-batches.' A loss term must be a differentiable function of the current parameters, but the embedding change that will result from the current gradient update is not available before that update is computed. It is therefore unclear whether SAR is a penalty on a reference embedding, a post-hoc update scaling, or something else, and the text does not explain how Eq. (4) is implemented. Moreover, λ in Eq. (4) is never given a value or a search range in Section 3.4 or Section 4.1, even though it controls the strength of the regularization. Without a precise algorithmic description and the λ value, the ablation RSS+SAR versus RSS cannot be interpreted and the method is not reproducible.","section":"Section 3.3, Eqs. (3)-(4); Section 3.4"},{"comment":"The inverse-staleness weighting in Eq. (2) is load-bearing but unvalidated. The paper asserts that 'features with smaller staleness will be covered with higher priorities because they might have higher possibilities of reappearing again in future datasets,' but no measurement of reappearance probability as a function of staleness is provided. Figure 6 shows only that SAS drops high-staleness features more, which verifies the algorithm implements the rule, not that the rule is correct. If reappearance probability is not monotonically decreasing in staleness, the reservoir may preserve the wrong samples. The authors should either measure P(reappear | staleness) on the datasets or compare SAS against an unweighted coverage baseline and a reversed-weight baseline to demonstrate that the monotone inverse weighting, rather than the coverage objective alone, drives the reported gains.","section":"Section 3.2, Eq. (2); Section 4.5"},{"comment":"The statistical reporting is insufficient for the size of the claimed effects. Table 2 reports p-values from a two-tailed pairwise t-test, but no standard deviations or confidence intervals are given. The text alternately says results are 'averaged over ten runs' and 'average scores over all the incremental datasets,' leaving it unclear whether the ten observations are time spans (which are sequential and not independent replicates) or random restarts. The absolute AUC differences are very small (for example, 0.0018 on Criteo against ASMG+SAS), so without variance information the reader cannot judge whether the rankings are stable. Please report standard deviations or confidence intervals, clarify the replication structure, and describe how the datasets are temporally split into D0 through D10.","section":"Section 4.1-4.2, Table 2"}],"minor_comments":[{"comment":"The sentence 'SAS will drop about 30% more stale features than the optimal solution. However, the SAS chooses the features with small staleness in prior and the actual drop ratio of stale features will be about 10%' is confusing; the relationship between the (1-1/e) approximation guarantee and the empirical 10% drop ratio should be explained more carefully.","section":"Section 3.2, after Theorem 1"},{"comment":"Algorithm 1 does not describe the neighbor-based optimization that the complexity analysis in Section 3.2 and Section 3.5 relies on; please include the neighbor update procedure or a pseudocode version of the optimized algorithm.","section":"Algorithm 1 and Section 3.2"},{"comment":"The temporal splitting of the datasets into D0 through D10 is not described; please specify how the 24 days of Criteo, the iPinYou logs, the Avazu data, and the 48 hours of Media are partitioned into pretraining and ten incremental periods.","section":"Section 4.1"},{"comment":"The notation for staleness uses s_i^t in Eq. (1) but plain s_i in Eq. (3); please align the notation to avoid ambiguity about the time index.","section":"Section 3.3, Eq. (3)"},{"comment":"Figure 5 plots Jaccard similarity and AUC with different scales; please clarify which y-axis corresponds to which quantity in the caption or legend.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"Dear Editor, the most serious issue is the inconsistency between the RI column and the AUC/Logloss values in Table 2, because it directly affects the headline 1.21% claim. I would ask the authors to reconcile these numbers and provide the raw per-time-span results. I would also request the exact value of λ and a precise algorithmic description of SAR, since the current text describes a loss term that depends on an update that has not yet been computed. These issues are addressable with additional analysis, so I recommend major revision rather than rejection. The code release and the breadth of datasets are positive signals, but the evidence as presented does not yet substantiate the main performance claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. FeSAIL is a legitimate, well-built incremental learning method for CTR prediction, and it deserves a serious referee. The new parts are SAS, which casts stale-feature replay as a weighted maximum coverage problem and solves it greedily with the standard (1-1/e) guarantee, and SAR, a staleness-scaled regularizer on embedding updates. Both are clean, and the neighbor-based complexity reduction makes SAS practical at scale. The experimental work is extensive: four datasets (including a private industrial one), ten time periods, multiple baselines, an ablation, runtime measurements, and pairwise t-tests. The code is public. That is more evidence than most CTR papers supply.\n\nThe weakest point is precisely what the stress-test flags: the inverse-staleness weighting in Eq. (2) rests on an assertion that features with smaller staleness are more likely to reappear. The paper never measures reappearance probability against staleness. Figure 6 just confirms the algorithm implements the priority rule; it does not validate the rule. This is a genuine gap in the causal story. It is not fatal: the sensitivity analysis in Figure 5 shows AUC is stable across a range of functions and biases, which suggests the method is not delicate in its weighting. But the paper would be much stronger with one plot showing that reappearance frequency actually decreases with staleness on these datasets.\n\nOther soft spots are more minor. The regularization weight lambda in Eq. (4) is not reported anywhere; the paper says grid search over some hyperparameters but omits this one. The private Media dataset adds real-world credibility but is not checkable. The reported gains are modest (0.78%-1.72% AUC over the best baseline), which is realistic for the field, but the t-test annotation is a good touch. I would not call the central claim in doubt; the method appears to work, and the math holds up.\n\nSo: send it to review. Ask the authors to validate the staleness-reappearance assumption and report lambda and other missing hyperparameters. This is a conditional accept rather than a reject. I'd cite it if I were doing replay-based continual learning.","headline":"FeSAIL is a solid, well-evaluated incremental CTR method with one unvalidated assumption about staleness and reappearance; worth refereeing and likely revisable.","tokens_in":13030,"tokens_out":2941,"would_cite":true,"duration_ms":28085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed-size reservoir of historical samples selected to cover recently stale features, plus a staleness-scaled regularizer, recovers most of the AUC lost when CTR models are incrementally retrained.","keywords":["CTR prediction","incremental learning","feature staleness","replay buffer","reservoir sampling","maximum coverage problem","embedding regularization","click-through rate"],"falsifier":"On the three public datasets, measure the conditional probability that a feature appears in the next incremental dataset given it has been absent for $s$ consecutive periods. If that probability is flat or increasing in $s$, the inverse-staleness weights in Eq. (2) prioritize the wrong samples, and the reported AUC gains should be re-tested against a reservoir chosen uniformly or by reappearance probability.","tokens_in":12072,"feed_emoji":"📈","tokens_out":13004,"duration_ms":110718,"temperature":0.7,"pith_summary":"CTR models retrained only on new data let embeddings of absent features go stale, and predictions on samples containing those features degrade when they reappear. FeSAIL counters this with two components: a staleness-aware sampler that fills a fixed-size reservoir by greedily covering the most valuable stale features, and a staleness-aware regularizer that scales embedding-update penalties by how long each feature has been absent. The paper reports that this combination outperforms six incremental-learning baselines on three public and one private CTR dataset, with an average relative AUC improvement of 1.21%. A sympathetic reader would take the central claim to be that replaying stale-feature samples with an inverse-staleness priority rule recovers most of the performance lost to feature staleness.","feed_headline":"Replay stale features: FeSAIL gains 1.21% AUC in click prediction","feed_subtitle":"A fixed-size reservoir plus staleness-scaled regularization keeps old embeddings in sync with fresh data.","key_machinery":"Two coupled mechanisms carry the argument. SAS maintains a staleness counter $s_i^t$ per feature, incremented when the feature is absent from the current incremental dataset and reset to 0 when it appears, and scores candidate replay samples by the total weight of stale features they cover, where weight decreases with staleness. The resulting selection is a greedy approximation to the maximum coverage problem, with neighbour-based updates reducing the per-step cost to $\\mathcal{O}(|\\mathcal{R}_t| m + \\bar{N} L m)$, where $\\bar{N}$ is the mean number of neighbours sharing a stale feature with the current pick. SAR adds the staleness-scaled guard term to the cross-entropy loss so the embedding update of a feature is damped in proportion to its staleness. Together they turn the reservoir into a fixed-size, staleness-targeted replay buffer and the training loss into a staleness-aware regularized objective.","core_discovery":"The central discovery is that feature staleness, measured as the number of consecutive time spans a feature is absent from the incremental data, can be handled at the replay-sampling and regularization levels without changing the CTR model architecture. FeSAIL first applies SAS, which treats filling the fixed reservoir as a maximum weighted coverage problem where each stale feature carries weight $w_i = \\mathrm{func}(s_i^t) + b$ decreasing in staleness, and solves it greedily with approximation ratio $1 - 1/e$. It then applies SAR, adding a guard term $g_i = \\frac{\\min(s_i, \\eta)}{\\min(s_{\\max}, \\eta)} \\|\\Delta e_i\\|^2$ to the loss so that embeddings of long-stale, low-frequency features can only change by small amounts. The paper reports that this combination outperforms incremental update, random sampling, RMFX, GAG, EWC, and ASMG on Criteo, iPinYou, Avazu, and a private Media dataset, with an average relative AUC improvement of 1.21% over the compared state-of-the-art methods.","pith_inferences":["A natural extension the paper does not pursue is to fit the weight function $w_i$ to empirically measured reappearance probabilities; if the true curve is steeper or flatter than inverse-proportional, the same budget $L$ could cover more of the features that actually return.","The SAS coverage objective counts features, not interaction recency, so it may under-weight a long-stale user-interest feature that is about to resurface; a recency-weighted variant is a plausible modification in settings with strong short-term drift.","The reported comparisons do not include a staleness-stratified random sampler of equal reservoir size, so the exact contribution of the greedy coverage step versus the fixed cap is not isolated; testing against that baseline would separate the two.","Because both components are model-agnostic and the guard term is non-parametric, FeSAIL transfers readily to other embedding-based continual learning problems, such as session-based recommendation, where the same absent-feature staleness appears."],"forward_implications":["FeSAIL is model-agnostic: it modifies only the sampling stage and the loss, so an Embedding&MLP-style CTR model can adopt it without architectural changes.","The fixed reservoir size keeps retraining cost predictable; the ablation on Avazu shows FeSAIL's sample count stays stable while full stale sampling grows with history.","The per-time-span results show FeSAIL maintaining its AUC advantage over the best baselines across most of the ten incremental periods, not just on average.","The two components are independently reusable: adding SAR to RMFX and GAG improves them, and adding SAS to EWC and ASMG improves them, which the paper presents as evidence that staleness awareness is compatible with other replay and regularization strategies."],"supporting_citations":[{"why":"Defines the practical incremental training setting for deep CTR models and the observation that absent-feature embeddings become stale; this is the problem FeSAIL targets.","marker":"[Wang et al., 2020]"},{"why":"Provides the base incremental meta-learning method used to illustrate the staleness effect and serves as the strongest baseline FeSAIL must beat.","marker":"[Peng et al., 2021]"},{"why":"Supplies the reservoir-replay and regularization conventions that SAS and SAR extend.","marker":"[Mi et al., 2020]"},{"why":"Establishes the memory-augmented reservoir replay framework for incremental session-based recommendation that FeSAIL's setting follows.","marker":"[Mi and Faltings, 2020]"},{"why":"Establishes the maximum coverage problem and the greedy approximation ratio $1 - 1/e$ that SAS inherits.","marker":"[Chekuri and Kumar, 2004]"},{"why":"RMFX baseline: an active-learning sample-based incremental recommender that FeSAIL compares against.","marker":"[Diaz-Aviles et al., 2012]"},{"why":"EWC baseline: the elastic weight consolidation approach whose regularization idea FeSAIL compares against.","marker":"[Kirkpatrick et al., 2016]"},{"why":"GAG baseline: a graph-based streaming session recommendation method used as a sample-based comparison in the experiments.","marker":"[Huang et al., 2020b]"}],"fun_headline_variants":["New method FeSAIL counters feature staleness in CTR models","Staleness-aware replay and regularization boost CTR prediction","Incremental CTR learning gets a staleness fix","FeSAIL tackles stale embeddings without model changes","Replay stale features to keep CTR embeddings fresh"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a feature absent for fewer time spans is more likely to reappear soon, so the reservoir should prioritize small-staleness features; the paper does not empirically measure reappearance probability as a function of staleness.","fun_headline_variants_meta":{"raw":{"variants":["New method FeSAIL counters feature staleness in CTR models","Staleness-aware replay and regularization boost CTR prediction","Incremental CTR learning gets a staleness fix","FeSAIL tackles stale embeddings without model changes","Replay stale features to keep CTR embeddings fresh"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1306,"prompt_tokens":971,"completion_tokens":335,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":259}},"tokens_in":587,"tokens_out":335,"duration_ms":3152,"temperature":1.0,"reasoning_tokens":259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:25:50.733970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the three public datasets, measure the conditional probability that a feature appears in the next incremental dataset given it has been absent for $s$ consecutive periods. If that probability is flat or increasing in $s$, the inverse-staleness weights in Eq. (2) prioritize the wrong samples, and the reported AUC gains should be re-tested against a reservoir chosen uniformly or by reappearance probability.","supporting_citations":[{"cited_title":"A Practical Incre- mental Method to Train Deep CTR Models","cited_arxiv_id":null,"evidence_quote":"Defines the practical incremental training setting for deep CTR models and the observation that absent-feature embeddings become stale; this is the problem FeSAIL targets."},{"cited_title":"Learning an Adaptive Meta Model- Generator for Incrementally Updating Recommender Sys- tems","cited_arxiv_id":null,"evidence_quote":"Provides the base incremental meta-learning method used to illustrate the staleness effect and serves as the strongest baseline FeSAIL must beat."},{"cited_title":"Memory Augmented Neural Model for Incremental Session-based Recommendation","cited_arxiv_id":null,"evidence_quote":"Establishes the memory-augmented reservoir replay framework for incremental session-based recommendation that FeSAIL's setting follows."},{"cited_title":"Maximum coverage problem with group budget con- straints and applications","cited_arxiv_id":null,"evidence_quote":"Establishes the maximum coverage problem and the greedy approximation ratio $1 - 1/e$ that SAS inherits."},{"cited_title":"Real- time top-n recommendation in social streams","cited_arxiv_id":null,"evidence_quote":"RMFX baseline: an active-learning sample-based incremental recommender that FeSAIL compares against."},{"cited_title":"Overcoming catastrophic forgetting in neural net- works","cited_arxiv_id":null,"evidence_quote":"EWC baseline: the elastic weight consolidation approach whose regularization idea FeSAIL compares against."}],"review_version":1}