{"id":"175243ec-eb75-49c2-824c-049273f22b62","arxiv_id":"2607.05908","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"Architectures with stronger inductive biases attain higher in-distribution accuracy but degrade faster under temporal distribution shift, while frozen pretrained encoders trade accuracy for stability.","lead":"This paper systematically compares how different neural network architectures degrade when data distributions shift over time, finding that models with stronger inductive biases achieve higher in-distribution accuracy but degrade faster. A smart generalist should read it because it provides practical, evidence-based guidance for selecting models in real-world systems where data evolves.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Frozen-encoder robustness confounds representation quality with trainable capacity: frozen backbones have 770–4k trainable params vs. 94k–2.2M for trained models, so lower decay may reflect limited head capacity rather than 'coarser, more stable representations.'","rationale":"The reader identifies the capacity confound among trained model families, which is valid but partially addressed by the existing data: the large variants (MLP-L, CNN-L, ResNet-L at ~2.1M params each) are approximately capacity-matched and the decay ordering still holds. The more severe and unaddressed confound is between representation quality and trainable capacity for frozen encoders: the ~100× gap in trainable parameters (770–4k vs. 94k–2.2M) provides an alternative explanation for their lower decay that the paper does not consider. This does not invalidate the paper's descriptive findings — the drift matrices are what they are — but it weakens the causal interpretation for frozen encoders, which is a load-bearing part of the practical guidance. The CONDITIONAL verdict remains appropriate: the empirical contribution is solid, the causal claim about why frozen encoders are robust needs the proposed check. The paper's reproducibility (code, W&B logs, versioned presets) and the breadth of the study (3 domains, 21+ models each) are genuine strengths that warrant credit. The within-family scaling data (small/medium/large variants) is an underutilized resource that could further disentangle capacity from inductive bias if analyzed explicitly.","tokens_in":40127,"tokens_out":3945,"duration_ms":292677,"concrete_test":"Train a linear classification head on frozen features from a randomly initialized ViT-S backbone (same architecture as DINOv2-S/DINOv3-S, but without any pretraining) on Yearbook, using the identical training protocol. If the 'random frozen' model's decay falls within the pretrained frozen encoder range (7.9–10.4%), then the robustness is driven by limited trainable capacity, not representation quality, and Pillar 2 of the central claim weakens. If it decays substantially more (e.g., >12%), representation quality contributes independently and the claim holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim has two pillars: (1) stronger inductive bias → faster decay among trained-from-scratch models, and (2) frozen pretrained encoders → slower decay due to 'coarser, more stable representations.' Pillar 1 is reasonably supported: the large variants (MLP-L, CNN-L, ResNet-L all at ~2.1M params; Table 3) are approximately capacity-matched, and the decay ordering holds (MLP-L: 12.2%, CNN-L: 13.7%, ResNet-L: 13.9%; Table 5). However, Pillar 2 has an unaddressed confound. The frozen encoders have 770–4k trainable parameters (Table 4), compared to 94k–2.2M for trained models (Table 3). A linear classification head cannot fit complex period-specific features regardless of backbone representation quality. The paper attributes the frozen encoders' lower decay to representation quality (§5.1: 'coarser, more stable representations'), but an equally plausible mechanism is that limited trainable capacity prevents overfitting to period-specific cues. The paper does not discuss this alternative. This matters because frozen-encoder robustness is central to the practical guidance (§6: 'architecture selection should weigh the expected horizon between retraining cycles'). If robustness stems from limited head capacity rather than representation quality, the guidance changes: one could achieve similar robustness by reducing trainable parameters on any backbone, without pretraining. Note: variation among frozen encoders (DINOv3-S: 7.9% decay vs. ViT-S16-IN21k: 10.4%) provides partial evidence that representation quality matters, but the gap between frozen and trained models (7.9–10.4% vs. 12.2–13.9%) could still be largely explained by the ~100× difference in trainable parameters.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"This paper presents a systematic empirical study of how neural architecture choices affect robustness to temporal distribution shift, evaluated across three time-indexed datasets (Yearbook, Amazon Reviews, arXiv) spanning image classification, text regression, and multi-label text classification. Using a unified temporal drift matrix framework, the authors train models on cumulative historical data and evaluate on both earlier and later periods, comparing model families from MLPs and CNNs to recurrent networks, Transformers, and frozen pretrained encoders. The central finding is that architectures with stronger inductive biases achieve higher in-distribution accuracy but degrade faster under temporal drift, while frozen pretrained encoders trade in-distribution accuracy for steadier temporal robustness. The study is well-motivated, the evaluation protocol is carefully designed, and the release of code, experiment presets, and W&B artifacts supports reproducibility.","tokens_in":41100,"tokens_out":1671,"duration_ms":389010,"significance":"The paper addresses a practically important question—how architectural inductive biases interact with temporal drift—that has not been systematically studied across multiple modalities and architecture families. The temporal drift matrix framework is a clean and reusable contribution, and the forgetting-curve and saliency-map analyses add qualitative depth. The reproducibility commitment is strong: versioned experiment presets, pinned environments, public W&B run histories, and per-model drift-matrix galleries. The cross-domain scope (image, text regression, multi-label text) and the breadth of architectures (12 trained-from-scratch plus 9 frozen encoders on Yearbook alone; 12 plus 7–8 on each text task) exceed prior work such as Wild-Time, which focused on a narrower architecture set per dataset. The practical guidance—that architecture selection should weigh expected retraining horizons, not just in-distribution accuracy—is actionable and well-supported by the data.","major_comments":[{"comment":"§5.1, Tables 3–5: The paper's central claim has two pillars. Pillar 1 (stronger inductive bias → faster decay among trained-from-scratch models) is reasonably supported: the large variants (MLP-L, CNN-L, ResNet-L at ~2.1M params; Table 3) are approximately capacity-matched, and the decay ordering holds (MLP-L: 12.2%, CNN-L: 13.7%, ResNet-L: 13.9%; Table 5). However, Pillar 2 (frozen pretrained encoders → slower decay due to 'coarser, more stable representations') has an unaddressed confound. The frozen encoders have 770–4k trainable parameters (Table 4) versus 94k–2.2M for trained models (Table 3). A linear classification head with 770 trainable parameters cannot fit complex period-specific features regardless of backbone representation quality. The paper attributes the frozen encoders' lower decay to representation quality (§5.1: 'coarser, more stable representations'), but an equallyPl","section":null},{"comment":"§5.1, Table 5: ViT-L (8.0% decay) is more robust than CNN-L (13.7%) and ResNet-L (13.9%), yet the paper's narrative groups ViTs with the 'strong inductive bias' models that 'degrade fastest.' The ViT family has the weakest spatial prior among trained models, so its lower decay is consistent with the paper's thesis, but the text does not discuss this explicitly. The omission matters because ViT-L's decay (8.0%) is closer to the frozen encoders (7.9–10.4%) than to the CNNs/ResNets, which complicates the binary framing of 'strong bias = fast decay' versus 'frozen encoder = slow decay.' The authors should clarify where ViTs fall in this spectrum and whether their weaker spatial prior explains their intermediate robustness.","section":null},{"comment":"§5.2.2, Table 21: On arXiv, the paper reports that all trained-from-scratch families have similar in-distribution AUC (97.2–98.1%) and similar decay (2.7–3.4%), and interprets this as evidence that 'where the bias gains nothing in distribution it forms no period-specific features to lose.' However, two frozen encoders (DeBERTa-v3: 6.5% decay, ELECTRA: 7.3%) degrade substantially more than the trained models. This is the opposite of the paper's general claim that frozen encoders are more robust. The paper mentions this anomaly but does not analyze it. Since DeBERTa-v3 is the most robust model on Amazon Reviews (0.043 decay; Table 13), its poor arXiv performance is puzzling and suggests that frozen-encoder robustness is dataset-dependent rather than driven by a general property of 'coarse, stable representations.' The authors should discuss this discrepancy and its implications for the un","section":null}],"minor_comments":[{"comment":"§4.3: The paper states that 'model selection is based on the final checkpoint.' This is unusual—most temporal robustness studies use early stopping on a validation set from the training period. Using the final checkpoint may overfit to the training period and inflate decay. A brief justification or a comparison with validation-based selection would strengthen the protocol.","section":null},{"comment":"§6.1: The limitations section acknowledges that families are not capacity-matched and that inductive bias and capacity are 'partially confounded,' but does not mention the frozen-encoder trainable-parameter disparity (770–4k vs. 94k–2.2M). This should be added for completeness.","section":null},{"comment":"Figure 2: The Amazon Reviews panel uses balanced MSE (lower is better), while Yearbook uses accuracy and arXiv uses macro AUC (both higher is better). The color scale direction is consistent within each panel, but a reader scanning across panels must mentally invert the interpretation. A note in the caption would help.","section":null},{"comment":"§3.1.2: The Amazon Reviews subsample is described as 'a stratified sample of 300,000 reviews' across seven categories, but the stratification variable (by category? by time? by rating?) is not specified. Clarifying this would aid reproduction.","section":null},{"comment":"Tables 6–9, 14–17, 22–25: Several per-cutoff tables show negative decay values (e.g., Table 8: CNN-L trained through 1978 has -5.4% decay, meaning future performance exceeds in-distribution performance). These cases are not discussed. A brief note explaining when and why this occurs would help interpretation.","section":null},{"comment":"§2.1, Figure 1: The figure caption references 'virtual drift' for covariate shift, citing [19], but [19] is a bachelor's thesis. While not incorrect, a more established reference for the virtual drift terminology would be preferable.","section":null},{"comment":"The paper mentions 'Extended version. Accepted at QCDS 2026; the proceedings version will appear in Springer LNCS.' If there are differences between this preprint and the accepted version, noting them would help reviewers assess novelty.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the frozen-encoder trainable-parameter confound is well-founded and is the primary reason for the major_revision recommendation. The paper's Pillar 2 claim is not adequately supported without addressing this alternative explanation. A simple control—e.g., training a linear head on a frozen randomly-initialized backbone, or reducing the trainable parameters of a trained-from-scratch model to match the frozen-encoder regime—would substantially strengthen or revise the claim. The paper is otherwise a solid empirical contribution with excellent reproducibility, and I would expect it to be acceptable after revision."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The three major comments each identify genuine gaps in the manuscript's interpretive framing, and we agree that all three warrant revision. Below we address each point in turn.","responses":[{"response":"The referee is correct that the trainable-parameter disparity is an unaddressed confound. The frozen encoders have 770–4k trainable parameters in their linear heads, while the trained-from-scratch models have 94k–2.2M. It is entirely plausible that the frozen encoders' lower decay is partly or wholly attributable to the limited capacity of the linear head to fit period-specific features, rather than (or in addition to) the stability of the pretrained backbone representations. Our manuscript currently attributes the lower decay solely to representation quality ('coarser, more stable representations'), which overstates what the experimental design can support. We will revise §5.1 and the abstract to acknowledge this confound explicitly. Specifically, we will add a paragraph in §5.1 noting that the frozen-encoder regime differs from the trained-from-scratch regime along two axes simultaneously—backbone representation quality and trainable head capacity—and that our experimental design does not isolate the two. We will soften the causal language from 'due to coarser, more stable representations' to something like 'associated with the frozen-encoder regime, which combines pretrained representations with substantially lower trainable capacity.' We will also note this as a limitation in §6.1 and flag the controlled experiment (e.g., matching head capacity across frozen and trained models) as future work. We cannot fully resolve this confound within the current experimental data, so the revision will be a candid acknowledgment rather than a new experiment.","revision_made":"yes","referee_comment":"§5.1, Tables 3–5: Pillar 2 (frozen pretrained encoders → slower decay due to 'coarser, more stable representations') has an unaddressed confound. The frozen encoders have 770–4k trainable parameters versus 94k–2.2M for trained models. A linear classification head with 770 trainable parameters cannot fit complex period-specific features regardless of backbone representation quality. The paper attributes the frozen encoders' lower decay to representation quality, but an equally plausible explanation is that the limited trainable head capacity mechanically prevents fitting period-specific features."},{"response":"We agree that the manuscript's narrative does not explicitly address where ViTs fall on the inductive-bias spectrum, and this omission creates an apparent tension with the binary framing of 'strong bias = fast decay' versus 'frozen encoder = slow decay.' The referee's observation is correct and important: ViTs have the weakest spatial prior among the trained-from-scratch models (self-attention over patches, no convolutional locality or translation equivariance), and ViT-L's decay of 8.0% is indeed closer to the frozen encoders (7.9–10.4%) than to the CNNs/ResNets (13.7–13.9%). This is actually consistent with our thesis—ViTs sit at the low-inductive-bias end of the trained-from-scratch spectrum, so their intermediate robustness is what the thesis predicts—but the manuscript does not make this explicit. We will revise §5.1 to discuss the ViT family's position explicitly. We will note that the inductive-bias spectrum among trained models runs from MLPs (no spatial prior) through ViTs (weak spatial prior via self-attention) to CNNs/ResNets (strong locality and translation equivariance), and that the decay ordering follows this gradient: ViT-L decays less than the CNNs/ResNets, consistent with its weaker spatial prior. We will also adjust the language to avoid the binary framing the referee rightly flags, replacing it with a spectrum-based description.","revision_made":"yes","referee_comment":"§5.1, Table 5: ViT-L (8.0% decay) is more robust than CNN-L (13.7%) and ResNet-L (13.9%), yet the paper's narrative groups ViTs with the 'strong inductive bias' models that 'degrade fastest.' The ViT family has the weakest spatial prior among trained models, so its lower decay is consistent with the paper's thesis, but the text does not discuss this explicitly. The omission matters because ViT-L's decay (8.0%) is closer to the frozen encoders (7.9–10.4%) than to the CNNs/ResNets. The authors should clarify where ViTs fall in this spectrum and whether their weaker spatial prior explains their intermediate robustness."},{"response":"The referee is correct that the DeBERTa-v3 and ELECTRA anomaly on arXiv is not adequately analyzed in the manuscript and that it complicates the unified claim. We will revise §5.2.2 to discuss this discrepancy explicitly. Our interpretation is as follows: on arXiv, the task (sorting papers into subject categories) is already well-solved by the shared frozen RoBERTa embeddings that all trained-from-scratch models use, so no architecture gains an in-distribution advantage and all trained models decay similarly (2.7–3.4%). The frozen encoders that perform worse (DeBERTa-v3, ELECTRA) likely do so not because their representations are less stable, but because their representations are less well-aligned with the arXiv task to begin with—DeBERTa-v3 and ELECTRA have lower in-distribution AUC on arXiv (90.7% and 88.9% at the 2000 cutoff; Table 22), so their higher decay reflects a lower starting point rather than faster forgetting of period-specific features. This is consistent with the referee's suggestion that frozen-encoder robustness is dataset-dependent. We will add this analysis to §5.2.2 and will also temper the general claim in the conclusion and abstract to note that frozen-encoder robustness is observed consistently on Yearbook and Amazon Reviews but does not hold uniformly on arXiv, where task-representation alignment moderates the effect. We acknowledge that we cannot fully explain why DeBERTa-v3, which is the most robust model on Amazon Reviews, is among the least robust on arXiv; the revision will state this as an open question rather than force a post-hoc rationalization.","revision_made":"yes","referee_comment":"§5.2.2, Table 21: On arXiv, two frozen encoders (DeBERTa-v3: 6.5% decay, ELECTRA: 7.3%) degrade substantially more than the trained models (2.7–3.4%). This is the opposite of the paper's general claim that frozen encoders are more robust. DeBERTa-v3 is the most robust model on Amazon Reviews (0.043 decay), so its poor arXiv performance is puzzling and suggests that frozen-encoder robustness is dataset-dependent rather than driven by a general property of 'coarse, stable representations.' The authors should discuss this discrepancy and its implications for the unified claim."}],"tokens_in":40302,"tokens_out":1474,"duration_ms":295453,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Two things to know: this is a well-executed empirical study that systematically compares temporal robustness across architectures and domains, and the central finding — that stronger inductive biases yield higher in-distribution accuracy but faster temporal decay — is supported by the data. The main soft spot is a confound in the frozen-encoder comparison that the paper doesn't address, though it doesn't break the core contribution.","headline":"Solid empirical study with one real confound in the frozen-encoder comparison","tokens_in":40978,"tokens_out":139,"would_cite":false,"duration_ms":131161,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Most accurate models degrade fastest under temporal drift","keywords":["temporal distribution shift","inductive bias","neural architecture","robustness","drift matrix","pretrained encoders","concept drift","model selection"],"falsifier":"If capacity-matched architectures with different inductive biases showed identical decay rates, the claim that inductive bias (rather than capacity) drives temporal robustness differences would be weakened.","tokens_in":40284,"feed_emoji":"⏳","tokens_out":690,"duration_ms":152108,"temperature":0.7,"pith_summary":"This paper presents a systematic empirical comparison of how different neural network architectures behave when data distributions shift over time. The central finding is that architectural inductive biases create a trade-off between in-distribution accuracy and temporal robustness. Models with strong structural priors, such as convolutional networks that exploit localized features, achieve the highest accuracy on held-out data from the training period, but those same features are the most period-specific and degrade fastest when the data drifts. Conversely, simpler models and frozen pretrained encoders that rely on coarser, more transferable representations start with lower accuracy but decay more gradually. The paper demonstrates this pattern across three domains — image classification, text regression, and multi-label text classification — using a unified evaluation framework called temporal drift matrices.","feed_headline":"Most accurate models degrade fastest under temporal drift","feed_subtitle":"Architectures that exploit sharp, localized features win in-distribution but lose them first as data evolves over time.","key_machinery":"The temporal drift matrix, a K×K grid where each cell records performance of a model trained on cumulative data through period i and evaluated on period j. Three summary statistics distill the matrix: in-distribution score (diagonal average), future score (average of cells where evaluation period exceeds training cutoff), and decay (the difference, oriented so positive always means worse robustness).","core_discovery":"The paper identifies an inverse relationship between in-distribution performance and temporal robustness that is governed by the strength of a model's inductive bias. Architectures with strong structural priors extract highly discriminative, localized features that are tightly bound to the training period and thus fragile under temporal shift. The paper also identifies a null case: on the arXiv multi-label classification task, where the pretrained representation already captures the discriminative signal, no architecture gains an in-distribution advantage, and consequently no architecture decays faster than the rest. This suggests the trade-off only manifests when an inductive bias actively扇","pith_inferences":[],"forward_implications":["Practitioners selecting architectures for deployment should weigh expected time-to-retraining alongside in-distribution accuracy, as the highest-scoring model at training time may be the least reliable once deployed.","Frozen pretrained encoders offer a middle ground: lower starting accuracy but steadier degradation, making them preferable when retraining cycles are long or infrequent.","The finding that the accuracy-robustness trade-off disappears when the task is already solved by the pretrained representation suggests that the trade-off is not inherent to architectures themselves but depends on whether the inductive bias extracts features beyond what the representation already provides."],"fun_headline_variants":["Strong inductive biases trade accuracy for temporal robustness","Models relying on localized features degrade fastest over time","In-distribution accuracy inversely linked to temporal stability","Pretrained encoders drift slower than feature-specific architectures","Temporal drift penalizes architectures that exploit sharp features"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper attributes robustness differences to inductive bias, but the model families are not capacity-matched. On Yearbook, smaller variants within each family decay less than larger ones, so model capacity — not just inductive bias — may be driving some of the observed decay patterns, leaving the central causal claim partially confounded.","fun_headline_variants_meta":{"raw":{"variants":["Strong inductive biases trade accuracy for temporal robustness","Models relying on localized features degrade fastest over time","In-distribution accuracy inversely linked to temporal stability","Pretrained encoders drift slower than feature-specific architectures","Temporal drift penalizes architectures that exploit sharp features","Sharp feature extraction accelerates model degradation under drift"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1153,"prompt_tokens":523,"completion_tokens":630,"prompt_tokens_details":null},"tokens_in":523,"tokens_out":630,"duration_ms":26037,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T20:43:53.615686+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If capacity-matched architectures with different inductive biases showed identical decay rates, the claim that inductive bias (rather than capacity) drives temporal robustness differences would be weakened.","supporting_citations":[],"review_version":1}