{"id":"1e5c568d-76be-4b6f-aa88-9266015aa6a0","arxiv_id":"2412.07273","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors introduce volatility cluster statistics (VCS) and a differentiable regularizer (VCA) to evaluate and reduce temporally clustered prediction errors in TGNNs, but the formal proof that AP/AU-ROC are blind to such patterns is incorrect.","lead":"This paper claims that standard metrics for temporal graph neural networks, like AP and AU-ROC, cannot detect when errors cluster in time. It introduces a new metric, VCS, and a training loss, VCA, and shows empirically that they reduce such error bursts, though the supporting theorem is flawed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1's proof assumes AP/AU-ROC depend only on binary disagreement counts, which is false for score-based metrics; the central claim is unsupported.","rationale":"The reader's weakest_assumption identifies exactly the flaw I find most load-bearing: the proof of Theorem 3.1 assumes away the score information that AP and AU-ROC use. My independent check via a simple counterexample confirms that two predictions with the same number of errors can yield different AP values, so the theorem cannot hold for the metrics it names. Since the paper's motivation, the definition of VCS/VCA, and the interpretation of the empirical results all depend on the conclusion that instance-based metrics are merely counting processes, this invalidates the central claim. The empirical study itself may contain useful observations, but the paper as written does not provide a sound theoretical basis for those observations; the missing code and unstated beta hyperparameter also hinder verification, though the theoretical flaw is sufficient for rejection. I agree with the reader's verdict and reasoning.","tokens_in":15537,"tokens_out":1896,"duration_ms":21996,"concrete_test":"Construct ground-truth Y=[1,0,1,0] and two predicted score vectors S1=[0.9,0.8,0.1,0.2] and S2=[0.6,0.7,0.4,0.3]. Under a 0.5 threshold, both give the same binary predictions [1,1,0,0], so H(Y, Y_hat1)=H(Y, Y_hat2)=2. Directly compute AP and AU-ROC for S1 and S2; they will differ (e.g., AP(S1) is higher because the positive scores are ranked above negatives). This refutes Theorem 3.1 as applied to AP/AU-ROC. If the authors instead restrict the theorem to hard-label metrics, then the theorem no longer covers the metrics used in the paper's own experiments.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central theoretical result, Theorem 3.1, claims that instance-based metrics cannot differentiate predictions with the same number of errors H(Y, Y_hat). The proof in Appendix A makes this true only by assuming f(y_i, y_hat_i) = c * 1[y_hat_i != y_i], i.e., the metric sees only binary correctness per sample. This assumption is false for AP and AU-ROC, which are functions of predicted scores (or ranks), not just of thresholded labels. Definition 2 allows f to be any function of y_i and y_hat_i, but the proof silently restricts y_hat_i to a hard label and f to an indicator. For actual TGNN outputs, y_hat_i is a probability; two score vectors with identical Hamming distance to the ground truth can have different AP/AU-ROC values. Therefore the theorem, as applied to the metrics it names, is invalid. The paper's motivating claim that these metrics 'reduce all diverse information to a mere disagreement count' fails, and the formal foundation for VCS/VCA is not established. The empirical findings about volatility clustering may be interesting and salvageable, but they do not rescue the unsound theoretical argument that justifies the paper's framing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies evaluation metrics for temporal graph neural networks (TGNNs), arguing that instance-based metrics such as AP and AU-ROC reduce a model's predictions to a mere disagreement count and therefore cannot capture temporal structures such as volatility clusters. It introduces a new metric, volatility-cluster statistics (VCS), inspired by Hopkins statistics, and a differentiable training objective, volatility-cluster-aware (VCA) learning, that regularizes against clustered errors. Experiments on five datasets and six TGNN models show that different TGNN families exhibit different temporal error patterns and that training with VCA reduces VCS values.","tokens_in":15779,"tokens_out":6794,"duration_ms":67157,"significance":"If the formal claim were correct, the paper would address a genuine gap: TGNN evaluation currently relies on AP and AU-ROC, which ignore temporal order, and the observed error-clustering patterns (memory-based models clustering at the end, RNN-based at the beginning, attention-based uniform) are interesting and potentially useful for model selection and system design. The VCS metric is intuitive and the empirical study is reasonably broad, covering five datasets and six models. However, the main theoretical result is invalid as stated, and the empirical contribution is weakened by the fact that VCA directly optimizes a smooth surrogate of VCS. The paper's core conclusion that AP and AU-ROC are merely counting processes is unsupported and in fact false for score-based metrics.","major_comments":[{"comment":"The proof of Theorem 3.1 assumes that f(y_i, ŷ_i) can be written as c·1[ŷ_i ≠ y_i], i.e., that an instance-based metric sees only binary correctness per sample. This is false for AP and AU-ROC, which are functions of predicted scores and their rankings. For example, with ground truth Y = [1, 0, 1, 0], the prediction score vectors (0.2, 0.1, 0.9, 0.3) and (0.9, 0.1, 0.4, 0.2) both have Hamming distance 1 from Y, but their AP values are 0.833 and 1, respectively. Thus Theorem 3.1 is false for the metrics it names, and the claimed 'counting process' characterization is not established.","section":"Section 3, Theorem 3.1 and Appendix A"},{"comment":"Definition 2 allows f to be any function of (y_i, ŷ_i), including one that returns the predicted score. The proof in Appendix A unilaterally restricts f to be c·1[ŷ_i ≠ y_i]. This restriction is not implied by link prediction being reducible to binary classification, since classification outputs are typically continuous scores. Without this restriction, the conclusion that all instance-based metrics depend only on the disagreement count does not follow.","section":"Section 3, Definition 2"},{"comment":"The VCA loss is a smooth and differentiable version of VCS itself. Consequently, the consistent decrease in VCS reported in Table 1 is a direct effect of optimizing an objective that includes that quantity. This demonstrates that the optimizer can trade off AP against VCS, but it does not by itself show that VCA captures a generalizable property beyond minimizing the metric it is defined from. The paper would need a comparison against alternative regularizers or an evaluation on metrics not included in the training objective to support the broader claim.","section":"Section 4, Eq. (4.4) and Table 1"},{"comment":"The main text defers technical proofs and experimental details to the extended arXiv version (Su and Wu 2024), which is a self-citation. The proof of Theorem 3.1 is actually contained in Appendix A of this submission, so the pointer is unnecessary and obscures the fact that the proof is present. Please reference the appendix within the submission rather than an external self-citation.","section":"Section 3 and Appendix A"}],"minor_comments":[{"comment":"In the text, 'Fig. 2(b) is the pattern for regular error' appears to be a typo; the caption labels (b) as 'Cluster Error' and (c) as 'Regular Error'.","section":"Section 3, Visualization Example"},{"comment":"References Xu et al. 2020a and 2020b are identical (both arXiv:2002.07962); one should be removed.","section":"References"},{"comment":"The model name 'JOIDE' appears to be a typo for 'JODIE'.","section":"Throughout"},{"comment":"Equation (4.4) uses a squared absolute difference of the VCS term, while Algorithm 3 uses an absolute value; please harmonize the notation.","section":"Section 4 and Algorithm 3"},{"comment":"The table reports means and standard deviations but no significance tests; several VCS reductions are within one standard deviation (e.g., TGAT on Reddit: 0.13±0.02 vs 0.10±0.02).","section":"Table 1"},{"comment":"Figure 4 shows variance for τ but the curves for γ have no error bars; please clarify whether these are single runs or averaged results.","section":"Figure 4"}],"recommendation":"reject","confidential_remarks":"The empirical observations about temporal error patterns are interesting and could perhaps be repurposed into a more modest empirical paper, but the central theoretical claim of this manuscript is load-bearing and false as stated. The self-citation to the extended arXiv version is also problematic, though it is not the basis for the rejection. I recommend rejection because the main formal result, Theorem 3.1, is incorrect for the very metrics (AP and AU-ROC) it claims to characterize, and the paper's framing cannot be repaired by a small patch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely useful empirical core, but the central theorem is wrong and the VCA training objective, as written, cannot train. It attacks a real gap: instance-based metrics like AP and AU-ROC throw away the temporal structure of errors, and volatility clustering is genuinely important for fault-tolerant and financial applications. VCS, a Hopkins statistic on the time axis, is a sensible complement to AP. The empirical observation that memory-based TGNNs cluster errors late in the test period while RNN-based models cluster them early is fresh and could be useful for model selection. The theory, however, is not sound. Theorem 3.1 claims any instance-based metric with the same Hamming distance to the ground truth must give the same value. The proof in Appendix A only gets that by assuming f(y_i, y_hat_i) = c * 1[y_hat_i != y_i]. That is not what AP or AU-ROC do. They are functions of the score or rank; you can have two predictions with the same number of thresholded errors and different AP. The theorem as stated is simply false, and it carries the paper's framing. That is not a side issue. I also have a serious problem with VCA. The regularizer in Eq. 4.4 is built on Tsoft, which uses the set Edisg of hard-thresholded disagreements. Even with the soft-min distance, the selection of Edisg is not differentiable with respect to the model parameters. As written, the regularizer supplies no gradient at all. The paper reports consistent VCS reductions, so either I am missing an approximation, or the experiments are not implementing what the text describes. The text does not tell us which, and beta is never reported. That makes the central empirical claim hard to trust. The experimental section is also missing a sanity check for VCS itself. There is no synthetic dataset with known ground-truth error patterns showing that VCS separates random, clustered, and regular patterns. Without that, the differences across model families could be an artifact of the metric. On the plus side, the authors clearly understand TGNNs and the applications, and the idea of a volatility-aware regularizer has potential. The paper is not nonsense, and the empirical observations, if they survive a proper check, would be a modest contribution to evaluation practice. But this version is not publishable. The false theorem needs to be removed or corrected, and the VCA gradient issue needs to be resolved or made explicit. If the empirical part holds up, a rewritten paper could be worth a real review. For this version, I would not accept it, and I would not cite it.","headline":"A useful empirical observation buried under an incorrect theorem and an unexplained training signal.","tokens_in":16268,"tokens_out":7629,"would_cite":false,"duration_ms":75781,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A proof that AP and AU-ROC can't see temporal error patterns, plus the metric that fixes it.","keywords":["temporal graph neural networks","evaluation metrics","volatility clustering","temporal error patterns","Hopkins statistic","temporal link prediction","regularization","instance-based metrics"],"falsifier":"Take a fixed sequence of correct/incorrect test labels with a fixed number of errors and shift the confidence scores among the errors (for instance, give the clustered errors higher scores); if AP or AU-ROC changes while the disagreement pattern stays the same, the theorem's reduction to a mere disagreement count fails.","tokens_in":15292,"feed_emoji":"⏱️","tokens_out":6938,"duration_ms":59630,"temperature":0.7,"pith_summary":"This paper argues that the standard metrics used to evaluate Temporal Graph Neural Networks (TGNNs) cannot distinguish between predictions that make the same number of errors but arrange those errors in different temporal patterns, such as bursts of failures. It proves this insensitivity in Theorem 3.1 and introduces a new statistic, Volatility-Cluster Statistics (VCS), adapted from Hopkins statistics, that measures how clustered prediction errors are in time. The paper also converts VCS into a differentiable training objective, Volatility-Cluster-Aware (VCA) learning, which reduces error clustering in TGNNs. A sympathetic reader should care because if the claim is right, current TGNN benchmarks systematically miss a failure mode that matters for fault-tolerant and financial systems.","feed_headline":"New metric catches error bursts that AP and AU-ROC miss","feed_subtitle":"A proof that standard scores miss clustered errors in temporal graph models—and a training fix.","key_machinery":"The central object is the Volatility-Cluster Statistics (VCS), built from a Hopkins-style ratio $T(E_{disg}, E_r) = D_r / (D_r + D_{disg})$ that compares the summed nearest-neighbour distances from randomly sampled times to the disagreement events against the same sum computed within the disagreement set. VCS is the absolute deviation $|1/2 - T|$ averaged over repeated random samples; a value near 0.5 indicates random timing of errors, near 1 indicates clustering, and near 0 indicates regular spacing. The training variant VCA replaces the non-differentiable nearest-neighbour distance with $d_{soft}(e, E) = -\\log(\\sum_{e' \\neq e} \\exp(-\\beta |t_e - t_{e'}|))/\\beta$, making the cluster penalty a differentiable regularizer that can be added to the prediction loss.","core_discovery":"The central claim is that any instance-based evaluation metric, defined per-sample and then aggregated, collapses all information about a model's predictions into a count of disagreements with ground truth. Two predictors that make the same number of mistakes but concentrate those mistakes at different times, randomly scattered versus grouped in a short window, receive identical AP and AU-ROC scores under this characterization. The paper formalizes this as Theorem 3.1 and demonstrates empirically that memory-based and RNN-based TGNNs produce distinctly clustered error patterns while attention-based models are more uniform, a difference standard metrics do not surface. It then establishes that the proposed VCS metric detects such clustering and that the VCA learning objective mitigates it, with reported experiments showing VCA reduces VCS by about 0.09 to 0.10 on average across datasets.","pith_inferences":["Editorial inference: the same insensitivity that VCS addresses applies to any evaluation that only counts hard errors, so a VCS-style correction could serve other temporal predictors such as traffic or anomaly detection, not just TGNNs.","Editorial inference: because AP and AU-ROC read score rankings, a metric that blends ranking with temporal clustering may capture more nuance than VCS's hard-error formulation; the paper does not explore this direction.","Editorial inference: VCS measures clustering in absolute clock time; an alternative worth testing is measuring in event-index or log-time space, which would make the statistic invariant to uneven event arrival rates."],"forward_implications":["If VCS is adopted, model selection among TGNNs can incorporate error timing rather than only error count, which matters for real-time systems that prefer uniformly distributed failures.","The paper's experiments show memory-based TGNNs (TGN, Tiger) cluster errors at the end of the test period while RNN-based models (JODIE, DyRep) cluster at the beginning, implying that the temporal mechanism of a TGNN shapes its error dynamics.","The reported results indicate that VCA regularization reduces VCS with little or no AP loss at a modest regularization strength, so mitigating error bursts does not necessarily sacrifice predictive accuracy.","Attention-based models already exhibit near-uniform error distributions in the paper's experiments, so VCA offers comparatively small improvements for that class of model."],"supporting_citations":[{"why":"Supplies the nearest-neighbour randomness statistic that VCS adapts to temporal error patterns.","marker":"Hopkins and Skellam 1954"},{"why":"Provides the cluster-validation formulation of the Hopkins statistic that the VCS definition follows.","marker":"Banerjee and Dave 2004"},{"why":"Benchmark showing near-identical AP across TGNNs that motivates the need for metrics with more discriminative power.","marker":"Poursafaei et al. 2022"},{"why":"Defines the temporal link prediction evaluation protocol and negative-sampling strategy used in the experiments.","marker":"Huang et al. 2024"},{"why":"The TGN model, a memory-based TGNN whose error-pattern behaviour is measured.","marker":"Rossi et al. 2021"},{"why":"The TGAT model, an attention-based TGNN whose more uniform error distribution is contrasted with memory- and RNN-based models.","marker":"Xu et al. 2020a"},{"why":"The JODIE model, an RNN-based TGNN exhibiting clustered errors early in the test period.","marker":"Kumar, Zhang, and Leskovec 2019"},{"why":"The DyRep model, an RNN-based TGNN evaluated in the VCS/VCA study.","marker":"Trivedi et al. 2019"},{"why":"The Tiger model, a memory-based TGNN whose clustered error pattern is reported.","marker":"Zhang et al. 2023"},{"why":"The TCL model, an attention-based TGNN used as a baseline with uniform error behaviour.","marker":"Wang et al. 2021b"}],"fun_headline_variants":["AP and AU-ROC miss error bursts—VCS catches them","Bursty errors invisible to AP/AU-ROC: new metric and training fix","Volatility clusters: the temporal graph blind spot AP/AU-ROC ignore","New VCS metric exposes error bursts standard scores can't see"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theorem that instance-based metrics see only disagreement counts assumes such a metric never looks at the model's numeric scores, only at whether each prediction is right or wrong.","fun_headline_variants_meta":{"raw":{"variants":["AP and AU-ROC miss error bursts—VCS catches them","Bursty errors invisible to AP/AU-ROC: new metric and training fix","Volatility clusters: the temporal graph blind spot AP/AU-ROC ignore","New VCS metric exposes error bursts standard scores can't see"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2564,"prompt_tokens":982,"completion_tokens":1582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":1511}},"tokens_in":598,"tokens_out":1582,"duration_ms":40552,"temperature":1.0,"reasoning_tokens":1511,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:56:29.489849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed sequence of correct/incorrect test labels with a fixed number of errors and shift the confidence scores among the errors (for instance, give the clustered errors higher scores); if AP or AU-ROC changes while the disagreement pattern stays the same, the theorem's reduction to a mere disagreement count fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the nearest-neighbour randomness statistic that VCS adapts to temporal error patterns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the cluster-validation formulation of the Hopkins statistic that the VCS definition follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Benchmark showing near-identical AP across TGNNs that motivates the need for metrics with more discriminative power."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the temporal link prediction evaluation protocol and negative-sampling strategy used in the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The TGN model, a memory-based TGNN whose error-pattern behaviour is measured."},{"cited_title":"TIGER: Temporal Interaction Graph Embedding with Restarts","cited_arxiv_id":"2302.06057","evidence_quote":"The Tiger model, a memory-based TGNN whose clustered error pattern is reported."}],"review_version":1}