{"id":"63a8dc33-ebbc-42bd-9164-1acbd3dffc92","arxiv_id":"2607.19382","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"On two French groundwater time series, attention-based replay sampling looks more structured and interpretable than random or loss-based alternatives, but the paper reports no numeric metrics to support its performance claims.","lead":"This paper explores whether explanation tools can reveal why adaptive forecasting models select certain past observations when learning from groundwater-level time series. It finds attention-based sample selection appears more interpretable than random or loss-based replay, but gives no quantitative proof of improved accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim in §5/§6 is unsupported: no error metric, baseline comparison, seed variance, or statistical test is reported for attention-guided replay, so 'improved predictive accuracy' cannot be verified.","rationale":"The paper's most consequential assertion is that attention-guided replay improves predictive accuracy under non-stationarity. That claim is load-bearing because the introduction, Section 5, and the conclusion all use it to motivate the paper's contribution. Yet the manuscript never reports quantitative forecasting performance. This is not a matter of disagreeing with an existing consensus or of stylistic preference: the evidence required to establish the claim is absent. The reader's verdict already flags the empirical claim as unsupported and assigns high correctness risk, so this stress-test does not move the verdict. I mark agreement as partial because the reader's formal 'weakest assumption' concerns the validity of attention-rollout/Grad-CAM attributions rather than the absence of accuracy metrics; however, the reader's rationale also mentions missing metrics and baselines, so the two readings are closely aligned. An additional concern about circularity (attention rollout both selects replay samples and is used to interpret those selections) reinforces the need for care, but the decisive issue is the missing quantitative comparison: without it, the central empirical claim cannot be accepted as established. The paper remains acceptable only as an exploratory workshop study or after adding the requested measurements and weakening the 'confirm' language.","tokens_in":6262,"tokens_out":5618,"duration_ms":50977,"concrete_test":"Obtain the numeric evaluation behind Figure 5, or rerun the authors' released code with the §4 hyperparameters (seq_len=180, pred_len=360, buffer_size=1500, freq=300, hidden_size=256, lr=1e-3, batch_size=16) on Piezometers 3 and 17. Compare forecasting errors (e.g., MSE or normalized RMSE) for random, max-loss, min-loss, attention-based replay, and a no-replay/static baseline across at least five random seeds, reporting mean±std and a paired significance test (e.g., Wilcoxon signed-rank). If attention replay does not significantly outperform the best baseline, the §5 accuracy claim is unsupported and should be removed or weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is in §5: 'the results confirm that attention-guided replay contributes to improved predictive accuracy under non-stationary conditions,' restated in §6. The manuscript provides no quantitative evidence for this claim. Section 4 fixes hyperparameters but no evaluation protocol; Section 5 contains no error metric (MSE, MAE, RMSE), no comparison of forecasting performance across replay strategies (random, max-loss, min-loss, attention), no static/no-replay baseline, no multiple seeds, no confidence intervals, and no statistical test. Figure 5 is described as 'forecasting performance' but no numeric values or an auditable caption are given, so the reader cannot verify the claimed improvement. Moreover, the attention-based strategy includes an additional sampling module with inner_loop=3 while the random/loss baselines do not; any observed gain would be confounded by added capacity/computation. The qualitative Grad-CAM and attention-rollout analyses (Figures 6-13) do not fill this gap because they are descriptive and never tied to the accuracy claim. Thus the central empirical assertion currently rests on an unverifiable visual claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates continual learning for long-term piezometric time-series forecasting under non-stationarity, comparing Experience Replay strategies (random, maximum-loss, minimum-loss, and attention-based) with PatchTST, PatchMixer, and DLinear. Explainability is used both to analyze model predictions (Grad-CAM) and to interpret replay-sample selection (attention rollout). The central claim is that attention-guided replay improves predictive accuracy during continual adaptation, while the qualitative analyses purport to show stable, model-agnostic sampling patterns and architecture-dependent temporal attribution. The manuscript is exploratory in tone, but it makes a direct empirical claim that is not currently supported by reported quantitative results.","tokens_in":6595,"tokens_out":2653,"duration_ms":27085,"significance":"If properly supported, the paper would provide a useful case study at the intersection of continual learning, time-series forecasting, and explainability for environmental monitoring. The qualitative observations about attention-based sampling and Grad-CAM attributions are plausible and could guide future work on interpretable replay selection. However, the main empirical claim—that attention-guided replay improves forecasting accuracy—rests on a single unquantified figure, and the explainability signals are used both as the mechanism and as the evidence for the mechanism's validity. The potential significance is therefore real but conditional on adding a rigorous quantitative evaluation and validating the attribution methods.","major_comments":[{"comment":"The central accuracy claim is not supported by any reported number. The text states in §5 that 'the results confirm that attention-guided replay contributes to improved predictive accuracy under non-stationary conditions,' and §6 repeats this, but Figure 5 is the only evidence and its caption gives no metric, no axis description, and no error bars. No MSE/MAE/RMSE values, no comparison against random, max-loss, min-loss, or no-replay baselines, no multiple seeds, no confidence intervals, and no statistical test are provided. In addition, the attention strategy includes an extra sampling module with inner_loop=3 while the other strategies do not, so any observed gain would be confounded by additional computation/capacity. This claim currently cannot be verified or reproduced.","section":"§5, Figure 5; §6"},{"comment":"The argument for stable, model-agnostic attention-based sampling is partly circular. Attention rollout is computed from the same sampling module used to select replay samples, so observing that attention-based sampling emphasizes high-attention regions is true to a large extent by construction. The claim that the sampling mechanism 'learns stable, model-agnostic criteria' is drawn from only two transformer/convolutional models and is not validated against any alternative importance measure or counterfactual test. I would recommend comparing attention rollout scores against input perturbation or leave-one-out importance, or at minimum showing that the selected samples are more informative than random selection with matched computational budget.","section":"§5, Figures 8–9 and 11; §4"},{"comment":"The Grad-CAM attributions are load-bearing for the qualitative conclusions about architectural differences (e.g., convolutional models use uniform evidence, transformers focus on recent time steps), yet no faithfulness or sanity check is reported. For sequence-to-sequence forecasting with a final linear layer of dimension pred_len, it is not obvious how gradient attributions are mapped back to the input time axis. Without validation against input perturbation, occlusion, or known synthetic data, the attributions may reflect artifacts of the model architecture rather than true predictive evidence. At minimum, the exact formulation of the Grad-CAM computation for these models, including the reshaping/upsampling step, should be specified and the conclusions should be correspondingly tempered.","section":"§4, Grad-CAM applied to final linear layer; §5, Figures 6–7"}],"minor_comments":[{"comment":"The computation of attention rollout for PatchMixer is not described. Since PatchMixer uses convolution rather than a standard transformer, clarify whether attention rollout is applied only to attention layers or to some generalized mixing operation.","section":"§4"},{"comment":"The figure caption says 'comparison applied on piezometer 3' but does not specify what is being compared or what the plotted quantity is. Add axis labels, metric definition, and ideally numerical values for the test set.","section":"§5, Figure 5"},{"comment":"The notion of 'top 25% highest-weighted data points, aggregated over time' is ambiguous. Is the threshold computed per figure, per model, per sequence, or globally? A formal definition would improve reproducibility.","section":"§5, Figures 6–9"},{"comment":"The role of inner_loop=3 in the attention-based sampling module is never explained. Does it correspond to multiple sampling iterations, gradient steps, or something else? This matters for the confound raised above.","section":"§4"},{"comment":"There are minor language issues (e.g., 'Some example with XAI has already shown promising results', 'Künstliche' should be 'Künstliche Intelligenz'). These do not affect content but should be corrected for a clean final version.","section":"Throughout"},{"comment":"No code or data availability statement is provided, and the dataset link is only in a footnote. Given the paper's empirical nature, a reproducibility statement with code and exact data splits would be valuable.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising exploratory angle, but the main empirical claim is currently unverifiable because no quantitative results are given. The circularity between the sampling criterion and the explainability evidence is a deeper concern, though it can be alleviated with validation experiments. I view this as repairable within the manuscript's scope, hence major revision rather than rejection. The editor may also want to ensure the authors provide the actual numerical evaluation during revision, since the lack of any error metric is unusual for a paper making a performance claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an honest exploratory study that overreaches in its abstract and §5. The qualitative XAI analysis of replay strategies on piezometric data is worth a look; the claim that attention-guided replay improves accuracy is not supported by any number in the paper.\n\nWhat's new: applying attention-based replay with Grad-CAM and attention rollout to two real groundwater time series, and comparing random/loss/attention sampling qualitatively. The authors do a decent job describing how the sampling patterns differ. The observation that PatchTST concentrates Grad-CAM attribution near the end of the input while PatchMixer spreads it out is a reasonable descriptive result. And the paper is candid about being exploratory.\n\nWhere it falls short: the central performance claim in §5 (“the results confirm that attention-guided replay contributes to improved predictive accuracy under non-stationary conditions”) appears without a single error metric, baseline comparator, seed variance, or statistical test. Figure 5 is a plot with no captioned numbers. On top of that, the attention strategy has an extra sampling module with inner_loop=3 while the other baselines do not, so even a visible difference would be confounded. The attention-rollout analysis is partly circular: the same attention scores drive sampling and then are offered as evidence that the sampler learns stable, model-agnostic criteria. That doesn't invalidate the observation, but it weakens the claim. Minor point: Grad-CAM on the final linear layer is a sensible choice but unvalidated, and the paper presents it as model reasoning.\n\nOverall: this is motivation, not evidence. A practitioner could use the qualitative insights to design a cleaner experiment. As a workshop paper, fine; as a claim, it overreaches. If it goes to peer review, the authors should add quantitative results with baselines and seeds, or drop the accuracy claim and frame the paper strictly as an XAI-for-CL exploratory study.","headline":"Exploratory XAI study overclaims the accuracy benefit; the qualitative observations are worth reading, but the central performance claim is unsupported by any reported number.","tokens_in":7017,"tokens_out":2538,"would_cite":false,"duration_ms":24930,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explainability signals — attention rollout and Grad-CAM — show that attention-guided replay improves accuracy on non-stationary groundwater forecasts and reveals how each model uses the past.","keywords":["continual learning","time series forecasting","explainability","attention rollout","Grad-CAM","experience replay","non-stationary data","groundwater level forecasting"],"falsifier":"Run the same continual-learning pipeline with replay samples chosen by the lowest attention-rollout scores instead of the highest; if predictive accuracy does not drop, attention is not the carrier of the reported gains. Likewise, mask the top Grad-CAM-attributed input steps and re-forecast: no error change would show the attributions are not causal.","tokens_in":6207,"feed_emoji":"💧","tokens_out":4587,"duration_ms":42976,"temperature":0.7,"pith_summary":"This paper argues that explainability tools can open the black box of continual learning for time series forecasting. It studies three forecasting architectures adapted with replay and knowledge distillation, and shows that an attention-based sampling strategy improves predictive accuracy on real groundwater-level series with regime shifts. Using attention rollout and Grad-CAM, it finds that sampling decisions follow stable, model-agnostic patterns, while prediction attributions expose different inductive biases: transformer models concentrate on recent time steps, convolutional models spread attention more evenly. The authors position explainability not as a final solution but as a diagnostic lens for understanding why replay works and when it can mislead.","feed_headline":"Attention-based replay improves non-stationary groundwater forecasts","feed_subtitle":"Why some historical steps get replayed — and how transformer and convolution models see the past differently.","key_machinery":"The explanatory engine is a pair of interpretability signals. Attention rollout, computed inside the attention-based replay sampler, assigns importance scores to temporal segments and directly drives which samples enter the replay buffer. Grad-CAM, applied to the final linear layer of the forecasting model, attributes each prediction back to input time steps. Together they connect the two levels of the system: why a sample is stored (sampling) and how the model uses stored patterns to forecast (attribution). The replay-plus-distillation continual learning setup (student trained from teacher on buffered samples) is the substrate on which these signals are measured.","core_discovery":"The paper's central claim is that attention-based Experience Replay not only helps forecasting models adapt to non-stationary groundwater data, but does so in a way that is interpretable. Attention rollout scores used to select replay samples yield consistent patterns across PatchMixer and PatchTST, suggesting the sampler learns model-agnostic notions of informative temporal segments. In contrast, Grad-CAM attribution maps show strong architectural differences: PatchTST focuses predictions on recent time steps while PatchMixer distributes attribution broadly. The paper also shows that random sampling is opaque, maximum-loss sampling over-emphasizes errors, and minimum-loss sampling becomes r","pith_inferences":["A natural extension the paper does not run: counterfactual deletion of attention-selected time steps to verify that Grad-CAM and rollout importance are causal for forecast error; if deletion tests fail, the explanations are correlational.","If attention-rollout patterns are stable, they might encode physical recharge or low-water events; that hypothesis could be tested against piezometer metadata such as drought indices without new model training.","The attention sampler's stability hints at a reusable 'data worthiness' measure across stations; a single pretrained sampler could flag which monitoring stations or periods deserve more frequent adaptation, reducing compute in large sensor networks.","A testable extension: use the variance of Grad-CAM attributions over time as a drift detector and compare its lead time against forecasting error rises."],"forward_implications":["If attention-guided replay indeed improves accuracy as reported, replay buffers for non-stationary forecasting can be built by scoring samples with attention rather than by loss or chance.","Grad-CAM's finding that PatchTST leans on recent observations and PatchMixer spreads attention predicts that each architecture will react differently to distribution shift; mixture or ensemble designs could compensate.","Stable, model-agnostic attention-rollout patterns imply an attention sampler trained with one forecasting architecture can likely be transferred to another without retraining.","The qualitative difference among sampling strategies suggests explainability should be an explicit criterion in replay-buffer design, not a post-hoc add-on.","Attribution maps could serve as early warnings of regime shifts in environmental monitoring if they change before forecast error jumps."],"fun_headline_variants":["Attention-based replay explains adaptive groundwater forecasting","Grad-CAM reveals model focus in continual learning forecasts","Why PatchTST and PatchMixer replay different history steps","Interpretable replay for non-stationary time series forecasting","Attention rollout uncovers informative past steps in forecasting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The analysis assumes the attention-rollout and Grad-CAM scores faithfully measure which input time steps are truly informative, and that the single piezometer case study represents the broader non-stationary setting.","fun_headline_variants_meta":{"raw":{"variants":["Attention-based replay explains adaptive groundwater forecasting","Grad-CAM reveals model focus in continual learning forecasts","Why PatchTST and PatchMixer replay different history steps","Interpretable replay for non-stationary time series forecasting","Attention rollout uncovers informative past steps in forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000411,"raw_usage":{"total_tokens":1937,"prompt_tokens":685,"completion_tokens":1252,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":429,"completion_tokens_details":{"reasoning_tokens":1176}},"tokens_in":429,"tokens_out":1252,"duration_ms":12061,"temperature":1.0,"reasoning_tokens":1176,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:13:08.328882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same continual-learning pipeline with replay samples chosen by the lowest attention-rollout scores instead of the highest; if predictive accuracy does not drop, attention is not the carrier of the reported gains. Likewise, mask the top Grad-CAM-attributed input steps and re-forecast: no error change would show the attributions are not causal.","supporting_citations":[],"review_version":1}