Pith. sign in

REVIEW 3 major objections 7 minor 43 references

Most accurate models degrade fastest under temporal drift

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 20:43 UTC pith:VFTD3AGH

load-bearing objection Solid empirical study with one real confound in the frozen-encoder comparison the 3 major comments →

arxiv 2607.05908 v1 pith:VFTD3AGH submitted 2026-07-07 cs.LG

Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift

Robin Holzinger (1) , Riccardo Colletti (1) ((1) Department of Electrical Engineering , Computer Sciences , University of California , Berkeley , USA) This is my paper
classification cs.LG
keywords temporal distribution shiftinductive biasneural architecturerobustnessdrift matrixpretrained encodersconcept driftmodel selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents a systematic empirical comparison of how different neural network architectures behave when data distributions shift over time. The central finding is that architectural inductive biases create a trade-off between in-distribution accuracy and temporal robustness. Models with strong structural priors, such as convolutional networks that exploit localized features, achieve the highest accuracy on held-out data from the training period, but those same features are the most period-specific and degrade fastest when the data drifts. Conversely, simpler models and frozen pretrained encoders that rely on coarser, more transferable representations start with lower accuracy but decay more gradually. The paper demonstrates this pattern across three domains — image classification, text regression, and multi-label text classification — using a unified evaluation framework called temporal drift matrices.

Core claim

The paper identifies an inverse relationship between in-distribution performance and temporal robustness that is governed by the strength of a model's inductive bias. Architectures with strong structural priors extract highly discriminative, localized features that are tightly bound to the training period and thus fragile under temporal shift. The paper also identifies a null case: on the arXiv multi-label classification task, where the pretrained representation already captures the discriminative signal, no architecture gains an in-distribution advantage, and consequently no architecture decays faster than the rest. This suggests the trade-off only manifests when an inductive bias actively扇

What carries the argument

The temporal drift matrix, a K×K grid where each cell records performance of a model trained on cumulative data through period i and evaluated on period j. Three summary statistics distill the matrix: in-distribution score (diagonal average), future score (average of cells where evaluation period exceeds training cutoff), and decay (the difference, oriented so positive always means worse robustness).

Load-bearing premise

The paper attributes robustness differences to inductive bias, but the model families are not capacity-matched. On Yearbook, smaller variants within each family decay less than larger ones, so model capacity — not just inductive bias — may be driving some of the observed decay patterns, leaving the central causal claim partially confounded.

What would settle it

If capacity-matched architectures with different inductive biases showed identical decay rates, the claim that inductive bias (rather than capacity) drives temporal robustness differences would be weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners selecting architectures for deployment should weigh expected time-to-retraining alongside in-distribution accuracy, as the highest-scoring model at training time may be the least reliable once deployed.
  • Frozen pretrained encoders offer a middle ground: lower starting accuracy but steadier degradation, making them preferable when retraining cycles are long or infrequent.
  • The finding that the accuracy-robustness trade-off disappears when the task is already solved by the pretrained representation suggests that the trade-off is not inherent to architectures themselves but depends on whether the inductive bias extracts features beyond what the representation already provides.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper presents a systematic empirical study of how neural architecture choices affect robustness to temporal distribution shift, evaluated across three time-indexed datasets (Yearbook, Amazon Reviews, arXiv) spanning image classification, text regression, and multi-label text classification. Using a unified temporal drift matrix framework, the authors train models on cumulative historical data and evaluate on both earlier and later periods, comparing model families from MLPs and CNNs to recurrent networks, Transformers, and frozen pretrained encoders. The central finding is that architectures with stronger inductive biases achieve higher in-distribution accuracy but degrade faster under temporal drift, while frozen pretrained encoders trade in-distribution accuracy for steadier temporal robustness. The study is well-motivated, the evaluation protocol is carefully designed, and the release of code, experiment presets, and W&B artifacts supports reproducibility.

Significance. The paper addresses a practically important question—how architectural inductive biases interact with temporal drift—that has not been systematically studied across multiple modalities and architecture families. The temporal drift matrix framework is a clean and reusable contribution, and the forgetting-curve and saliency-map analyses add qualitative depth. The reproducibility commitment is strong: versioned experiment presets, pinned environments, public W&B run histories, and per-model drift-matrix galleries. The cross-domain scope (image, text regression, multi-label text) and the breadth of architectures (12 trained-from-scratch plus 9 frozen encoders on Yearbook alone; 12 plus 7–8 on each text task) exceed prior work such as Wild-Time, which focused on a narrower architecture set per dataset. The practical guidance—that architecture selection should weigh expected retraining horizons, not just in-distribution accuracy—is actionable and well-supported by the data.

major comments (3)
  1. §5.1, Tables 3–5: The paper's central claim has two pillars. Pillar 1 (stronger inductive bias → faster decay among trained-from-scratch models) is reasonably supported: the large variants (MLP-L, CNN-L, ResNet-L at ~2.1M params; Table 3) are approximately capacity-matched, and the decay ordering holds (MLP-L: 12.2%, CNN-L: 13.7%, ResNet-L: 13.9%; Table 5). However, Pillar 2 (frozen pretrained encoders → slower decay due to 'coarser, more stable representations') has an unaddressed confound. The frozen encoders have 770–4k trainable parameters (Table 4) versus 94k–2.2M for trained models (Table 3). A linear classification head with 770 trainable parameters cannot fit complex period-specific features regardless of backbone representation quality. The paper attributes the frozen encoders' lower decay to representation quality (§5.1: 'coarser, more stable representations'), but an equallyPl
  2. §5.1, Table 5: ViT-L (8.0% decay) is more robust than CNN-L (13.7%) and ResNet-L (13.9%), yet the paper's narrative groups ViTs with the 'strong inductive bias' models that 'degrade fastest.' The ViT family has the weakest spatial prior among trained models, so its lower decay is consistent with the paper's thesis, but the text does not discuss this explicitly. The omission matters because ViT-L's decay (8.0%) is closer to the frozen encoders (7.9–10.4%) than to the CNNs/ResNets, which complicates the binary framing of 'strong bias = fast decay' versus 'frozen encoder = slow decay.' The authors should clarify where ViTs fall in this spectrum and whether their weaker spatial prior explains their intermediate robustness.
  3. §5.2.2, Table 21: On arXiv, the paper reports that all trained-from-scratch families have similar in-distribution AUC (97.2–98.1%) and similar decay (2.7–3.4%), and interprets this as evidence that 'where the bias gains nothing in distribution it forms no period-specific features to lose.' However, two frozen encoders (DeBERTa-v3: 6.5% decay, ELECTRA: 7.3%) degrade substantially more than the trained models. This is the opposite of the paper's general claim that frozen encoders are more robust. The paper mentions this anomaly but does not analyze it. Since DeBERTa-v3 is the most robust model on Amazon Reviews (0.043 decay; Table 13), its poor arXiv performance is puzzling and suggests that frozen-encoder robustness is dataset-dependent rather than driven by a general property of 'coarse, stable representations.' The authors should discuss this discrepancy and its implications for the un
minor comments (7)
  1. §4.3: The paper states that 'model selection is based on the final checkpoint.' This is unusual—most temporal robustness studies use early stopping on a validation set from the training period. Using the final checkpoint may overfit to the training period and inflate decay. A brief justification or a comparison with validation-based selection would strengthen the protocol.
  2. §6.1: The limitations section acknowledges that families are not capacity-matched and that inductive bias and capacity are 'partially confounded,' but does not mention the frozen-encoder trainable-parameter disparity (770–4k vs. 94k–2.2M). This should be added for completeness.
  3. Figure 2: The Amazon Reviews panel uses balanced MSE (lower is better), while Yearbook uses accuracy and arXiv uses macro AUC (both higher is better). The color scale direction is consistent within each panel, but a reader scanning across panels must mentally invert the interpretation. A note in the caption would help.
  4. §3.1.2: The Amazon Reviews subsample is described as 'a stratified sample of 300,000 reviews' across seven categories, but the stratification variable (by category? by time? by rating?) is not specified. Clarifying this would aid reproduction.
  5. Tables 6–9, 14–17, 22–25: Several per-cutoff tables show negative decay values (e.g., Table 8: CNN-L trained through 1978 has -5.4% decay, meaning future performance exceeds in-distribution performance). These cases are not discussed. A brief note explaining when and why this occurs would help interpretation.
  6. §2.1, Figure 1: The figure caption references 'virtual drift' for covariate shift, citing [19], but [19] is a bachelor's thesis. While not incorrect, a more established reference for the virtual drift terminology would be preferable.
  7. The paper mentions 'Extended version. Accepted at QCDS 2026; the proceedings version will appear in Springer LNCS.' If there are differences between this preprint and the accepted version, noting them would help reviewers assess novelty.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The three major comments each identify genuine gaps in the manuscript's interpretive framing, and we agree that all three warrant revision. Below we address each point in turn.

read point-by-point responses

  1. Referee: §5.1, Tables 3–5: Pillar 2 (frozen pretrained encoders → slower decay due to 'coarser, more stable representations') has an unaddressed confound. The frozen encoders have 770–4k trainable parameters versus 94k–2.2M for trained models. A linear classification head with 770 trainable parameters cannot fit complex period-specific features regardless of backbone representation quality. The paper attributes the frozen encoders' lower decay to representation quality, but an equally plausible explanation is that the limited trainable head capacity mechanically prevents fitting period-specific features.

    Authors: The referee is correct that the trainable-parameter disparity is an unaddressed confound. The frozen encoders have 770–4k trainable parameters in their linear heads, while the trained-from-scratch models have 94k–2.2M. It is entirely plausible that the frozen encoders' lower decay is partly or wholly attributable to the limited capacity of the linear head to fit period-specific features, rather than (or in addition to) the stability of the pretrained backbone representations. Our manuscript currently attributes the lower decay solely to representation quality ('coarser, more stable representations'), which overstates what the experimental design can support. We will revise §5.1 and the abstract to acknowledge this confound explicitly. Specifically, we will add a paragraph in §5.1 noting that the frozen-encoder regime differs from the trained-from-scratch regime along two axes simultaneously—backbone representation quality and trainable head capacity—and that our experimental design does not isolate the two. We will soften the causal language from 'due to coarser, more stable representations' to something like 'associated with the frozen-encoder regime, which combines pretrained representations with substantially lower trainable capacity.' We will also note this as a limitation in §6.1 and flag the controlled experiment (e.g., matching head capacity across frozen and trained models) as future work. We cannot fully resolve this confound within the current experimental data, so the revision will be a candid acknowledgment rather than a new experiment. revision: yes

  2. Referee: §5.1, Table 5: ViT-L (8.0% decay) is more robust than CNN-L (13.7%) and ResNet-L (13.9%), yet the paper's narrative groups ViTs with the 'strong inductive bias' models that 'degrade fastest.' The ViT family has the weakest spatial prior among trained models, so its lower decay is consistent with the paper's thesis, but the text does not discuss this explicitly. The omission matters because ViT-L's decay (8.0%) is closer to the frozen encoders (7.9–10.4%) than to the CNNs/ResNets. The authors should clarify where ViTs fall in this spectrum and whether their weaker spatial prior explains their intermediate robustness.

    Authors: We agree that the manuscript's narrative does not explicitly address where ViTs fall on the inductive-bias spectrum, and this omission creates an apparent tension with the binary framing of 'strong bias = fast decay' versus 'frozen encoder = slow decay.' The referee's observation is correct and important: ViTs have the weakest spatial prior among the trained-from-scratch models (self-attention over patches, no convolutional locality or translation equivariance), and ViT-L's decay of 8.0% is indeed closer to the frozen encoders (7.9–10.4%) than to the CNNs/ResNets (13.7–13.9%). This is actually consistent with our thesis—ViTs sit at the low-inductive-bias end of the trained-from-scratch spectrum, so their intermediate robustness is what the thesis predicts—but the manuscript does not make this explicit. We will revise §5.1 to discuss the ViT family's position explicitly. We will note that the inductive-bias spectrum among trained models runs from MLPs (no spatial prior) through ViTs (weak spatial prior via self-attention) to CNNs/ResNets (strong locality and translation equivariance), and that the decay ordering follows this gradient: ViT-L decays less than the CNNs/ResNets, consistent with its weaker spatial prior. We will also adjust the language to avoid the binary framing the referee rightly flags, replacing it with a spectrum-based description. revision: yes

  3. Referee: §5.2.2, Table 21: On arXiv, two frozen encoders (DeBERTa-v3: 6.5% decay, ELECTRA: 7.3%) degrade substantially more than the trained models (2.7–3.4%). This is the opposite of the paper's general claim that frozen encoders are more robust. DeBERTa-v3 is the most robust model on Amazon Reviews (0.043 decay), so its poor arXiv performance is puzzling and suggests that frozen-encoder robustness is dataset-dependent rather than driven by a general property of 'coarse, stable representations.' The authors should discuss this discrepancy and its implications for the unified claim.

    Authors: The referee is correct that the DeBERTa-v3 and ELECTRA anomaly on arXiv is not adequately analyzed in the manuscript and that it complicates the unified claim. We will revise §5.2.2 to discuss this discrepancy explicitly. Our interpretation is as follows: on arXiv, the task (sorting papers into subject categories) is already well-solved by the shared frozen RoBERTa embeddings that all trained-from-scratch models use, so no architecture gains an in-distribution advantage and all trained models decay similarly (2.7–3.4%). The frozen encoders that perform worse (DeBERTa-v3, ELECTRA) likely do so not because their representations are less stable, but because their representations are less well-aligned with the arXiv task to begin with—DeBERTa-v3 and ELECTRA have lower in-distribution AUC on arXiv (90.7% and 88.9% at the 2000 cutoff; Table 22), so their higher decay reflects a lower starting point rather than faster forgetting of period-specific features. This is consistent with the referee's suggestion that frozen-encoder robustness is dataset-dependent. We will add this analysis to §5.2.2 and will also temper the general claim in the conclusion and abstract to note that frozen-encoder robustness is observed consistently on Yearbook and Amazon Reviews but does not hold uniformly on arXiv, where task-representation alignment moderates the effect. We acknowledge that we cannot fully explain why DeBERTa-v3, which is the most robust model on Amazon Reviews, is among the least robust on arXiv; the revision will state this as an open question rather than force a post-hoc rationalization. revision: yes

Circularity Check

0 steps flagged

No circularity found — empirical study with no derivations or fitted constants

full rationale

This is a purely empirical study with no theoretical derivations, fitted constants, or first-principles predictions that could be circular. The temporal drift matrix (Eq. 1) is defined as M_ij = perf(f_i, D_j), a direct evaluation of trained models on held-out data. The summary statistics ID, Fut, and Dec (Eqs. 6–8) are computed directly from matrix entries with no fitting. The central claim—that architectural inductive biases shape temporal robustness—is supported by empirical observations on three external benchmarks (Yearbook, Amazon Reviews, arXiv) using standard pretrained models, not by any self-cited theoretical result. The two self-citations (Ref [4] Modyn, Ref [19] Holzinger thesis) are contextual references to retraining orchestration and drift analysis infrastructure; neither is load-bearing for the paper's claims about inductive bias and temporal decay. The paper explicitly acknowledges confounds (capacity not matched, §6.1) rather than hiding them behind a derivation. No step in the evaluation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper is an empirical study with no new theoretical entities or postulated constructs. Free parameters are standard hyperparameters pinned in versioned configs. Axioms are standard domain assumptions from the distribution shift literature, with one ad-hoc assumption about capacity matching that the paper itself flags as a limitation.

free parameters (3)
  • Learning rates = task-specific
    Described as task-specific and pinned in versioned presets, not fitted to the target result.
  • Number of epochs = fixed
    Fixed number of epochs with model selection on final checkpoint, not tuned per model.
  • Class weights (w_c, w_r) = inverse frequency
    Computed from data statistics, not free parameters.
axioms (4)
  • standard math Temporal distribution shift can be decomposed into covariate, label, and concept drift.
    Following established terminology in concept drift research [11], used to frame the problem in §2.1.
  • domain assumption Cumulative historical training reflects realistic deployment scenarios.
    Stated in §4.1 to justify the cumulative training strategy D≤k.
  • domain assumption Model selection based on the final checkpoint is representative of best obtainable performance.
    Stated in §4.3: 'model selection is based on the final checkpoint.'
  • ad hoc to paper Architectures within a family at different sizes isolate the effect of inductive bias.
    The paper acknowledges in §6.1 that families are not capacity-matched, so this assumption is partially violated.

pith-pipeline@v1.1.0-glm · 42969 in / 2140 out tokens · 540878 ms · 2026-07-08T20:43:53.615686+00:00 · methodology

0 comments
read the original abstract

Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood. We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains encompassing image classification, multi-label text classification, and text regression tasks. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross-temporal generalization. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders. Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually. These observations offer practical guidance for selecting architectures for real-world systems subject to temporal drift.

Figures

Figures reproduced from arXiv: 2607.05908 by Berkeley, Computer Sciences, Riccardo Colletti (1) ((1) Department of Electrical Engineering, Robin Holzinger (1), University of California, USA).

Figure 1
Figure 1. Figure 1: Drift types in a binary classification setting. Cir [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cohort-mean drift matrices, one panel per domain. Each cell [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Forgetting curves on Yearbook. Each curve is one model’s mean accuracy as the gap between its training year and the evaluation year grows, averaged over training years and seeds. A zero gap is in-distribution, and the slope is the rate of forgetting (Section C.7). The forgetting curves in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Gradient saliency maps on Yearbook for CNN-L, ResNet-S, and MLP-L, each trained through 1950 (left) and 1970 (right) and evaluated on later years. Each panel averages gradient saliency over the selected portraits from the indicated evaluation year rather than a single example. by token and the Transformers through self-attention, and so capture how a review’s words compose into its rating. That fit shows u… view at source ↗
Figure 5
Figure 5. Figure 5: Cohort-mean Accuracy matrix M¯ over the Yearbook models. Cell (i, j) is the mean across those models of the score from training through slice i and evaluating on slice j. C.1. Model Roster [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: MLP models: Accuracy drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. C.3. CNN 1905 1915 1928 1937 1946 1955 1964 1973 1982 1991 2000 2009 2013 Evaluation year 1905 1915 1928 1937 1946 1955 1964 1973 1982 1991 2000 2009 2013 Training year 50 60 70 80 90 Accuracy (%) 1905 1915 1928 1937 1946 1955 196… view at source ↗
Figure 7
Figure 7. Figure 7: CNN models: Accuracy drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: ResNet models: Accuracy drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. C.5. ViT 1905 1915 1928 1937 1946 1955 1964 1973 1982 1991 2000 2009 2013 Evaluation year 1905 1915 1928 1937 1946 1955 1964 1973 1982 1991 2000 2009 2013 Training year 50 60 70 80 90 Accuracy (%) 1905 1915 1928 1937 1946 1955 … view at source ↗
Figure 9
Figure 9. Figure 9: ViT models: Accuracy drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Transfer models: Accuracy drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Forgetting curves: each model (left) and averaged within each family (right). [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Models ranked by mean future performance and by temporal decay. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Cohort-mean Balanced MSE matrix M¯ over the Amazon Reviews models. Cell (i, j) is the mean across those models of the score from training through slice i and evaluating on slice j. D.1. Model Roster [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: FFN models: Balanced MSE drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. D.3. TextCNN 2014-H1 2015-H1 2016-H1 2017-H1 2018-H1 2019-H1 2020-H1 2021-H1 2022-H1 2023-H1 2023-H2 Evaluation half-year 2014-H1 2015-H1 2016-H1 2017-H1 2018-H1 2019-H1 2020-H1 2021-H1 2022-H1 2023-H1 2023-H2 Training half-y… view at source ↗
Figure 15
Figure 15. Figure 15: TextCNN models: Balanced MSE drift matrix [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Recurrent models: Balanced MSE drift matrix [PITH_FULL_IMAGE:figures/full_fig_p023_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Transformer models: Balanced MSE drift matrix [PITH_FULL_IMAGE:figures/full_fig_p023_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Frozen models: Balanced MSE drift matrix [PITH_FULL_IMAGE:figures/full_fig_p024_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Forgetting curves: each model (left) and averaged within each family (right). [PITH_FULL_IMAGE:figures/full_fig_p025_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Models ranked by mean future performance and by temporal decay. [PITH_FULL_IMAGE:figures/full_fig_p025_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Cohort-mean Macro AUC matrix M¯ over the arXiv models. Cell (i, j) is the mean across those models of the score from training through slice i and evaluating on slice j. E.1. Model Roster [PITH_FULL_IMAGE:figures/full_fig_p027_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: FFN models: Macro AUC drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. E.3. TextCNN 2000 2003 2006 2009 2012 2015 2018 2021 2024 2025 Evaluation year 2000 2003 2006 2009 2012 2015 2018 2021 2024 2025 Training year 88 90 92 94 96 98 Macro AUC (%) 2000 2003 2006 2009 2012 2015 2018 2021 2024 2025 Eva… view at source ↗
Figure 23
Figure 23. Figure 23: TextCNN models: Macro AUC drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Recurrent models: Macro AUC drift matrix [PITH_FULL_IMAGE:figures/full_fig_p029_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Transformer models: Macro AUC drift matrix [PITH_FULL_IMAGE:figures/full_fig_p029_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Frozen models: Macro AUC drift matrix M(m) and deviation from the cohort mean ∆(m) = M(m) − M¯ for each model, shown on a sequential and a zero-centred diverging scale, respectively. E.7. Forgetting and Rankings To see how quickly each model forgets, we summarize its drift matrix as a forgetting curve. The curve plots the Macro AUC against the lag ℓ = j − i, the number of slices between the training cutof… view at source ↗
Figure 27
Figure 27. Figure 27: Forgetting curves: each model (left) and averaged within each family (right). [PITH_FULL_IMAGE:figures/full_fig_p031_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Models ranked by mean future performance and by temporal decay. [PITH_FULL_IMAGE:figures/full_fig_p031_28.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages · 6 internal anchors

  1. [1]

    Automatically detecting data drift in machine learning classifiers

    Samuel Ackerman, Orna Raz, Marcel Zalmanovici, and Aviad Zlotnick. Automatically detecting data drift in machine learning classifiers, 2021. URL https: //arxiv.org/abs/2111.05672

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  2. [2]

    Aguiar and Alberto Cano

    Gabriel J. Aguiar and Alberto Cano. A comprehen- sive analysis of concept drift locality in data streams,

    work page

  3. [3]

    URL https://arxiv.org/abs/2311. 06396

    work page

  4. [4]

    A theory of learning from different domains

    Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151–175, 2010

    work page 2010

  5. [5]

    Modyn: Data-centric machine learning pipeline orchestration.Proc

    Maximilian Böther, Ties Robroek, Viktor Gsteiger, Robin Holzinger, Xianzhe Ma, Pınar Tözün, and Ana Klimovic. Modyn: Data-centric machine learning pipeline orchestration.Proc. ACM Manag. Data, 3 (1), February 2025. doi: 10.1145/3709705. URL https://doi.org/10.1145/3709705

    work page doi:10.1145/3709705 2025

  6. [6]

    Learning

    Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger 9 Drift Happens: Neural Architecture Robustness to Temporal Distribution Shift Schwenk, and Yoshua Bengio. Learning phrase rep- resentations using RNN encoder–decoder for statisti- cal machine translation. InProceedings of the 2014 Conference on Empirical Meth...

    work page doi:10.3115/v1/d14-1179 2014

  7. [7]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V . Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR), 2020

    work page 2020

  8. [8]

    arxiv dataset, 2024

    Cornell University. arxiv dataset, 2024. URL https: //www.kaggle.com/dsv/7548853

    work page arXiv 2024

  9. [9]

    BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL- HLT), pages 4171–4186. Association for Computa- tio...

    work page doi:10.18653/v1/n19-1423 2019

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations (ICLR), 2021

    work page 2021

  11. [11]

    EV A-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171, 2024

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171, 2024

    work page 2024

  12. [12]

    A survey on concept drift adaptation.ACM Computing Surveys, 46(4):1–37, 2014

    João Gama, Indr ˙e Žliobait ˙e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. A survey on concept drift adaptation.ACM Computing Surveys, 46(4):1–37, 2014. doi: 10.1145/2523813

    work page doi:10.1145/2523813 2014

  13. [13]

    A century of portraits: A visual historical record of american high school yearbooks

    Shiry Ginosar, Kate Rakelly, Sarah Sachs, Brian Yin, and Alexei A Efros. A century of portraits: A visual historical record of american high school yearbooks. InIEEE International Conference on Computer Vision Workshops, pages 1–7, 2015

    work page 2015

  14. [14]

    Shiry Ginosar, Kate Rakelly, Sarah Sachs, Brian Yin, Crystal Lee, Philipp Krahenbuhl, and Alexei A. Efros. A century of portraits: A visual historical record of american high school yearbooks, 2019

    work page 2019

  15. [15]

    In search of lost domain generalization

    Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. InInternational Confer- ence on Learning Representations, 2021

    work page 2021

  16. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    work page 2016

  17. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), 2022

    work page 2022

  18. [18]

    De- BERTaV3: Improving DeBERTa using ELECTRA- style pre-training with gradient-disentangled embed- ding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. De- BERTaV3: Improving DeBERTa using ELECTRA- style pre-training with gradient-disentangled embed- ding sharing. InInternational Conference on Learning Representations (ICLR), 2023

    work page 2023

  19. [19]

    Long short- term memory.Neural Computation, 9(8):1735–1780,

    Sepp Hochreiter and Jürgen Schmidhuber. Long short- term memory.Neural Computation, 9(8):1735–1780,

    work page

  20. [20]

    doi: 10.1162/neco.1997.9.8.1735

    work page doi:10.1162/neco.1997.9.8.1735 1997

  21. [21]

    An analysis of drift- and cost-aware ml retraining triggering policies in modyn

    Robin Holzinger. An analysis of drift- and cost-aware ml retraining triggering policies in modyn. Bachelor’s thesis, Technical University of Munich, September

    work page

  22. [22]

    Supervisors: Prof. Dr. Viktor Leis., Jana Vatter, Prof. Dr. Ana Klimovi´c, Maximilian Böther

    work page

  23. [23]

    Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders

    Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation.arXiv preprint arXiv:2403.03952, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Convolutional neural networks for sen- tence classification

    Yoon Kim. Convolutional neural networks for sen- tence classification. InProceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1746–1751. As- sociation for Computational Linguistics, 2014. doi: 10.3115/v1/D14-1181

    work page doi:10.3115/v1/d14-1181 2014

  25. [25]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Confer- ence on Learning Representations, 2015

    work page 2015

  26. [26]

    WILDS: A benchmark of in-the-wild distribution shifts

    Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubra- mani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. WILDS: A benchmark of in-the-wild distribution shifts. InInternational Confer- ence on Machine Learning, pages 5637–5664. PMLR, 2021

    work page 2021

  27. [27]

    RoBERTa: A Robustly Optimized BERT Pretraining Approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke 10 Drift Happens: Neural Architecture Robustness to Temporal Distribution Shift Zettlemoyer, and Veselin Stoyanov. RoBERTa: A ro- bustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1907

  28. [28]

    A Con- vNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A Con- vNet for the 2020s. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2022

    work page 2022

  29. [29]

    Cost- effective retraining of machine learning models, 2023

    Ananth Mahadevan and Michael Mathioudakis. Cost- effective retraining of machine learning models, 2023

    work page 2023

  30. [30]

    Cost-aware retraining for machine learning

    Ananth Mahadevan and Michael Mathioudakis. Cost-aware retraining for machine learning. Knowledge-Based Systems, 293:111610, 2024. ISSN 0950-7051. doi: https://doi.org/10. 1016/j.knosys.2024.111610. URL https: //www.sciencedirect.com/science/ article/pii/S0950705124002454

    work page arXiv 2024

  31. [31]

    DINOv2: Learning robust visual features without su- pervision.Transactions on Machine Learning Re- search (TMLR), 2024

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Woj- ciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patri...

    work page 2024

  32. [32]

    Failing loudly: An empirical study of methods for detecting dataset shift

    Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. InAdvances in Neural Information Processing Systems, volume 32, 2019

    work page 2019

  33. [33]

    Learning trans- ferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning trans- ferable visual models from natural language supervi- sion. InProceedings of the 38th International Confer- ence on Machine Learning (ICML), 2021

    work page 2021

  34. [34]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  35. [35]

    Oriane Siméoni, Huy V . V o, Maximilian Seitzer, Fed- erico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ra- mamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Cou- prie,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    MPNet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. MPNet: Masked and permuted pre-training for language understanding. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

  37. [37]

    Con- tinuum: A platform for cost-aware, low-latency con- tinual learning

    Huangshi Tian, Minchen Yu, and Wei Wang. Con- tinuum: A platform for cost-aware, low-latency con- tinual learning. InProceedings of the ACM Sympo- sium on Cloud Computing, SoCC ’18, page 26–40, New York, NY , USA, 2018. Association for Com- puting Machinery. ISBN 9781450360111. doi: 10.1145/3267809.3267817. URL https://doi. org/10.1145/3267809.3267817

    work page doi:10.1145/3267809.3267817 2018

  38. [38]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc., 2017

    work page 2017

  39. [39]

    MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MiniLM: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2020

    work page 2020

  40. [40]

    Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, mem- ory efficient, and long context finetuning and inference. ...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  41. [41]

    Hellerstein, Natacha Crooks, and Joseph E

    Sarah Wooders, Xiangxi Mo, Amit Narang, Kevin Lin, Ion Stoica, Joseph M. Hellerstein, Natacha Crooks, and Joseph E. Gonzalez. Ralf: Accuracy-aware scheduling for feature store maintenance.Proc. VLDB Endow., 17(3):563–576, nov 2023. ISSN 2150-8097. doi: 10.14778/3632093.3632116. URL https:// doi.org/10.14778/3632093.3632116

    work page doi:10.14778/3632093.3632116 2023

  42. [42]

    Wild-time: A benchmark of in-the-wild distribution shift over time

    Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. InAdvances in Neural Information Processing Sys- tems, 2022

    work page 2022

  43. [43]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), 2023. 11 Drift Happens: Neural Architecture Robustness to Temporal Distribution Shift Appendix A. Reproducibility Experiments are defined as versioned preset s...

    work page 2023