pith. sign in

arxiv: 2508.03941 · v4 · submitted 2025-08-05 · 💻 cs.IR · cs.LG

Measuring the stability and plasticity of recommender systems

Maria Jo\~ao Lavoura , Robert Jungnickel , Jo\~ao Vinagre This is my paper

Pith reviewed 2026-05-18 23:49 UTC · model grok-4.3

classification 💻 cs.IR cs.LG
keywords recommender systemsstabilityplasticitytemporal evaluationmodel retrainingoffline protocollong-term behavior
0
0 comments X

The pith

Recommender models can be profiled by their stability in retaining old patterns versus plasticity in adapting to new ones when retrained over time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an evaluation approach that tracks how recommendation algorithms perform across repeated retraining steps on time-ordered data. Traditional tests only check performance at one moment, but live systems keep changing as fresh interactions appear. The new protocol measures stability as the capacity to maintain accuracy on earlier patterns and plasticity as the speed of adjusting to emerging ones. This matters for picking models that will hold up as user behavior shifts rather than excelling only in a single snapshot. The method works across any dataset, algorithm family, or accuracy measure and is demonstrated with early results on book-interaction data.

Core claim

Recommendation models display characteristic stability and plasticity profiles when they are retrained sequentially on successive temporal portions of interaction data. Stability captures how well a model continues to match patterns from earlier periods, while plasticity captures how rapidly it incorporates newer patterns. An offline protocol built around repeated retraining produces these profiles in a manner independent of any particular dataset, algorithm, or performance metric. Experiments with three algorithm types on the GoodReads collection indicate that different techniques yield distinct profiles and point toward a possible trade-off between the two properties.

What carries the argument

The stability-plasticity profiling protocol that retrains models on successive time-based splits of interaction data and tracks retention of past patterns alongside adaptation to newer ones.

If this is right

  • Models can be compared and chosen according to whether an application requires stronger retention of established patterns or faster response to new ones.
  • Different algorithmic families produce distinguishable stability-plasticity signatures rather than uniform behavior.
  • Long-term evaluation becomes possible instead of relying solely on single-point accuracy scores.
  • A trade-off may exist such that gains in stability reduce plasticity and vice versa.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The protocol could be used to test whether stability-plasticity balance changes when models are updated at different frequencies.
  • Results on additional datasets would clarify whether the observed algorithmic differences hold beyond book-interaction data.
  • Integrating the offline profiles with live monitoring could help detect when a deployed model begins to lose either property.

Load-bearing premise

Sequential retraining on temporal splits of an offline dataset sufficiently reproduces the pattern shifts that occur in live online recommender systems.

What would settle it

Deploy the same set of models in a live recommender system, record their stability and plasticity rankings from real user interactions, and check whether those rankings match the order produced by the offline temporal-split protocol.

Figures

Figures reproduced from arXiv: 2508.03941 by Jo\~ao Vinagre, Maria Jo\~ao Lavoura, Robert Jungnickel.

Figure 1
Figure 1. Figure 1: illustrates this setup in a model vs holdout, as well as the comparisons (arrows) we associate to stability and plasticity. To compute stability and plasticity metrics, we collect four test￾ing scores corresponding to the performance of models 𝑀1 and 𝑀2 on 𝐷 𝑇 𝑒𝑠𝑡 1 and 𝐷 𝑇 𝑒𝑠𝑡 2 . We denote these scores as 𝑆1,1 and 𝑆1,2 corresponding to 𝑀1 and 𝑆2,1 and 𝑆2,2 corresponding to 𝑀2. D1 Test D2 Test M1 M2 S1,1 … view at source ↗
Figure 2
Figure 2. Figure 2: UserKNN model accuracy scores Figures 2, 3 and 4 show the HitRatio@20 scores for each combi￾nation. By looking at these result heatmaps, we can already spot some important differences between algorithms. For example, it is 1https://cseweb.ucsd.edu/~jmcauley/datasets/goodreads.html 2plus interactions from the first day of 2015 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: BPRMF model accuracy scores [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: NeuMF model accuracy scores Algorithm Stability Plasticity UKNN 1.038 0.180 BPRMF 0.989 0.283 NeuMF 1.008 0.276 [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
read the original abstract

The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of this dataset to train a model, and the remaining data to measure how closely the model recommendations match the observed user interactions. This protocol is straightforward, useful and practical, but it only provides snapshot performance. We know, however, that online systems evolve over time. In general, it is a good idea that models are frequently retrained with recent data. But if this is the case, to what extent can we trust previous evaluations? How will a model perform when a different pattern (re)emerges? In this paper we propose a methodology to study how recommendation models behave when they are retrained. The idea is to profile algorithms according to their ability to, on the one hand, retain past patterns - stability - and, on the other hand, (quickly) adapt to changes - plasticity. We devise an offline evaluation protocol that provides detail on the long-term behavior of models, and that is agnostic to datasets, algorithms and metrics. To illustrate the potential of this framework, we present preliminary results of three different types of algorithms on the GoodReads dataset that suggest different stability and plasticity profiles depending on the algorithmic technique, and a possible trade-off between stability and plasticity. We further discuss the potential and limitations of the proposal and advance some possible improvements.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes an offline evaluation protocol to profile recommender algorithms by stability (retention of past patterns under sequential retraining) and plasticity (adaptation to new patterns) on temporal splits of static datasets such as GoodReads. It illustrates the approach with preliminary results for three algorithm types that suggest distinct profiles and a possible stability-plasticity trade-off, claiming the framework is agnostic to datasets, algorithms, and metrics and provides insight into long-term behavior beyond snapshot evaluations.

Significance. If validated, the framework would address a genuine gap in recommender-systems evaluation by moving beyond static hold-out protocols to characterize temporal dynamics. The dataset-, algorithm-, and metric-agnostic design is a positive feature that could enable systematic comparisons across techniques.

major comments (2)
  1. [Proposed Methodology / Evaluation Protocol] The central claim that sequential retraining on temporal partitions of offline logs (e.g., GoodReads) produces distribution shifts representative of live systems is load-bearing for the entire methodology. Offline static data lack closed-loop feedback in which recommendations alter subsequent user behavior; this endogenous non-stationarity is absent, risking systematic underestimation of plasticity and overestimation of stability for models sensitive to recommendation-induced shifts.
  2. [Preliminary Results / Experiments] The preliminary results paragraph asserts that the three algorithm types exhibit different stability-plasticity profiles and a possible trade-off, yet supplies no equations, exact metric definitions, quantitative values, tables, or figures. Without these, it is impossible to verify whether the observed differences support the claimed profiles or trade-off.
minor comments (1)
  1. [Abstract / Introduction] The abstract and introduction repeatedly use the terms 'stability' and 'plasticity' before they are formally defined; a brief forward reference to the precise definitions would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address the two major comments point by point below, clarifying our position on the offline protocol and committing to specific revisions that strengthen the presentation without overclaiming equivalence to live systems.

read point-by-point responses

  1. Referee: The central claim that sequential retraining on temporal partitions of offline logs (e.g., GoodReads) produces distribution shifts representative of live systems is load-bearing for the entire methodology. Offline static data lack closed-loop feedback in which recommendations alter subsequent user behavior; this endogenous non-stationarity is absent, risking systematic underestimation of plasticity and overestimation of stability for models sensitive to recommendation-induced shifts.

    Authors: We agree that static offline logs cannot reproduce closed-loop feedback, where recommendations themselves shape future interactions and thereby induce endogenous non-stationarity. Our protocol deliberately uses temporal partitions of existing static data to create observable distribution shifts in a reproducible, dataset-agnostic manner; it therefore measures stability and plasticity under exogenous pattern changes rather than claiming to replicate the full dynamics of a live recommender. We will revise the manuscript to (i) state this scope limitation explicitly in the introduction and methodology sections, (ii) add a dedicated limitations paragraph that acknowledges the risk of underestimating plasticity for feedback-sensitive models, and (iii) suggest future extensions that incorporate simulated or online closed-loop data. These changes temper the central claim while preserving the utility of the offline framework as a practical first step. revision: partial

  2. Referee: The preliminary results paragraph asserts that the three algorithm types exhibit different stability-plasticity profiles and a possible trade-off, yet supplies no equations, exact metric definitions, quantitative values, tables, or figures. Without these, it is impossible to verify whether the observed differences support the claimed profiles or trade-off.

    Authors: The manuscript contains a dedicated experimental section that defines the stability and plasticity metrics via explicit formulas, reports numerical results on GoodReads, and includes tables and figures comparing the three algorithm families. To address the referee’s concern that these details are not sufficiently visible in the high-level summary, we will expand the preliminary-results paragraph (and the corresponding experimental subsection) to restate the metric equations, insert representative quantitative values, and add explicit cross-references to the tables and figures that illustrate the distinct profiles and the observed stability-plasticity trade-off. revision: yes

Circularity Check

0 steps flagged

New offline evaluation protocol for stability/plasticity is defined independently without reduction to fitted inputs or self-citations

full rationale

The paper introduces a methodology for profiling recommender systems by stability (retention of past patterns) and plasticity (adaptation to changes) via sequential retraining on temporal splits of offline datasets. This framework is presented as a direct definition of the two concepts applied to retraining behavior, with no equations, fitted parameters, or load-bearing self-citations that reduce the central claim to its own inputs by construction. The protocol is explicitly described as agnostic to datasets, algorithms, and metrics, and the preliminary results on GoodReads are illustrative rather than definitional. No step in the provided text equates a derived quantity to a prior fit or renames an input as a prediction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The proposal rests on the domain assumption that temporal splits of interaction data can proxy real-world pattern evolution; no free parameters or invented entities are introduced in the abstract.

axioms (1)
  • domain assumption Temporal splits of a static dataset can be used to simulate the evolution of user-item interaction patterns in live systems.
    The protocol description relies on this to create successive training and test periods without stating independent validation of the simulation fidelity.

pith-pipeline@v0.9.0 · 5782 in / 1246 out tokens · 35629 ms · 2026-05-18T23:49:50.251880+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · 1 internal anchor

  1. [1]

    Marie Al-Ghossein, Talel Abdessalem, and Anthony Barré. 2022. A Survey on Stream-Based Recommender Systems.ACM Comput. Surv.54, 5 (2022), 104:1– 104:36. https://doi.org/10.1145/3453443

    work page doi:10.1145/3453443 2022

  2. [2]

    Robert M French. 1999. Catastrophic forgetting in connectionist networks.Trends in cognitive sciences3, 4 (1999), 128–135

    work page 1999

  3. [3]

    João Gama. 2012. A survey on learning from data streams: current and future trends.Prog. Artif. Intell.1, 1 (2012), 45–55. https://doi.org/10.1007/S13748-011- 0002-6

    work page doi:10.1007/s13748-011- 2012

  4. [4]

    Ihsan Günes, Cihan Kaleli, Alper Bilge, and Huseyin Polat. 2014. Shilling attacks against recommender systems: a comprehensive survey.Artif. Intell. Rev.42, 4 (2014), 767–799. https://doi.org/10.1007/S10462-012-9364-9

    work page doi:10.1007/s10462-012-9364-9 2014

  5. [5]

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural Collaborative Filtering. InProceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (Eds.). ACM, 173–182. https://doi.org/10.1145/3038912.3052569

    work page doi:10.1145/3038912.3052569 2017

  6. [6]

    Hayes, and Christo- pher Kanan

    Ronald Kemker, Marc McClure, Angelina Abitino, Tyler L. Hayes, and Christo- pher Kanan. 2018. Measuring Catastrophic Forgetting in Neural Networks. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Adva...

    work page doi:10.1609/aaai.v32i1.11651 2018

  7. [7]

    David Lopez-Paz and Marc’Aurelio Ranzato. 2017. Gradient Episodic Memory for Continual Learning. InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Decem- ber 4-9, 2017, Long Beach, CA, USA, Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwana...

    work page 2017

  8. [8]

    Pawel Matuszyk, João Vinagre, Myra Spiliopoulou, Alípio Mário Jorge, and João Gama. 2018. Forgetting techniques for stream-based matrix factorization in Measuring the stability and plasticity of recommender systems Conference’17, July 2017, Washington, DC, USA recommender systems.Knowl. Inf. Syst.55, 2 (2018), 275–304. https://doi.org/10. 1007/S10115-017-1091-8

    work page 2018

  9. [9]

    Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. 2013. The stability- plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects.Frontiers in psychology4 (2013), 54654

    work page 2013

  10. [10]

    Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme

    work page

  11. [11]

    InUAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009, Jeff A

    BPR: Bayesian Personalized Ranking from Implicit Feedback. InUAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009, Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452–461. https://www.auai.org/uai2009/papers/UAI2009_0139_ 48141db02b9f0b02bc7158819ebfa2c7.pdf

    work page 2009

  12. [12]

    Three scenarios for continual learning

    Gido M. van de Ven and Andreas S. Tolias. 2019. Three scenarios for continual learning.CoRRabs/1904.07734 (2019). arXiv:1904.07734 http://arxiv.org/abs/ 1904.07734

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  13. [13]

    João Vinagre and Alípio Mário Jorge. 2012. Forgetting mechanisms for scalable collaborative filtering.J. Braz. Comput. Soc.18, 4 (2012), 271–282. https://doi. org/10.1007/S13173-012-0077-3

    work page doi:10.1007/s13173-012-0077-3 2012

  14. [14]

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2024. A Comprehensive Survey of Continual Learning: Theory, Method and Application.IEEE Trans. Pattern Anal. Mach. Intell.46, 8 (2024), 5362–5383. https://doi.org/10.1109/TPAMI. 2024.3367329

    work page doi:10.1109/tpami 2024

  15. [15]

    Jose, Beibei Kong, and Yudong Li

    Fajie Yuan, Guoxiao Zhang, Alexandros Karatzoglou, Joemon M. Jose, Beibei Kong, and Yudong Li. 2021. One Person, One Model, One World: Learning Continual User Representation without Forgetting. InSIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, Fernando Di...

    work page doi:10.1145/3404835.3462884 2021

  16. [16]

    Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual Learning Through Synaptic Intelligence. InProceedings of the 34th International Confer- ence on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 3987–3995. http://proceedings....

    work page 2017

  17. [17]

    Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2021. RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. InCIKM ’21: Th...

    work page doi:10.1145/3459637.3482016 2021