{"id":"e21b3d64-bd67-4716-8afa-794027345902","arxiv_id":"2607.27861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Model rankings and module-ablation conclusions on CTDG benchmarks flip with the negative-candidate count and distribution, so the paper recommends full-catalog (all-entity) ranking as the primary evaluation protocol.","lead":"This paper shows that on four standard temporal-graph benchmarks, which model is 'best' depends on how many random wrong answers it is scored against, and on how those wrong answers are chosen. It argues that benchmarks should rank against the full fixed catalog of destinations, and shows that part of sampled-metric performance comes from a seen-versus-unseen shortcut.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The model-order reversals and module-effect sign flips rely on a non-standard unified BCE training recipe and only 3 seeds; retraining with original recipes and more seeds may eliminate the flips, weakening the empirical basis of the central claim.","rationale":"The paper's theoretical derivation is sound: Eq. (3) correctly identifies the density-ratio score, Eq. (7) correctly computes expected finite-K reciprocal rank, and the 2x2 membership diagnostics are well designed. The empirical demonstration, however, is the load-bearing evidence for the central claim that published model-superiority conclusions are conditional. That demonstration depends on a unified 3-epoch BCE training recipe that is not the original recipe for most of the six models, and on only three seeds. The paper itself shows that the Wikipedia module-effect sign flip reverses under BPR, which is direct evidence that at least one headline conclusion is training-recipe-dependent. Since no CI or per-seed analysis is given for the model-pair order reversals, the reader cannot tell whether the reversals are consistent or noise. The reader's weakest assumption focused on the fixed-catalog scope; I agree that is a scope limitation, but the paper explicitly acknowledges it. The training-recipe/seed concern is more threatening because it targets the empirical basis of the central claim itself, not just its scope. The reader listed objective sensitivity as a condition, but did not make it the primary weakness; my concern is that the broader model-order reversals may also be objective-sensitive. If retraining with original recipes removes the reversals, the paper's central claim would reduce to a theoretical possibility plus a demonstration on an artificial training setup, which would warrant a stronger condition or even rejection of the empirical generalization. Because the theory remains intact and the code is promised but unexecuted, the existing CONDITIONAL verdict is appropriate, with the additional condition that the reversals be verified across original training recipes and more seeds.","tokens_in":13644,"tokens_out":11993,"duration_ms":118024,"concrete_test":"Using the authors' public repository, retrain all six models under their original training recipes (original loss, epochs, hyperparameters) with at least 10 seeds. For each seed and each dataset, compute Expected-20 and full-catalog MRR for every model and determine whether the specific pair reversals reported in Table V (CRAFT/DyGFormer on LastFM; leading-three order on MOOC; CRAFT/CRAFT-R on Wikipedia) occur. Require the reversal to appear in at least 8 of 10 seeds for the dataset-level claim to hold. Also compute seed-paired 95% CIs for the model-pair MRR differences under each protocol; if the Expected-20 and full-catalog CIs overlap or the reversal CI includes zero, the empirical evidence for a robust conditional ordering is weak.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that model-superiority conclusions from sampled-negative benchmarks are conditional on q and K—rests on the observation that at least one model pair reverses between Expected-20 and full-catalog order on LastFM, MOOC, and Wikipedia. But all six models are trained with a single non-standard recipe: 3 epochs, BCE, one random negative per positive. Table IX shows that the Wikipedia CRAFT/CRAFT-R module-effect sign flip reverses when the original BPR objective is used; this raises the possibility that the model-order reversals in Table V are artifacts of the unified training setup rather than a property of the published benchmarks. Moreover, only three seeds are reported. The mean ± SD for the flipping pairs (e.g., LastFM CRAFT .4283±.0112 vs DyGFormer .4161±.0015 at Expected-20; CRAFT .0636±.0065 vs DyGFormer .0811±.0100 full) leaves open the possibility that the reversal is not consistent across seeds. The paper provides seed-paired CIs for module effects but not for model-pair order differences, so the headline 'at least one pair flips on three datasets' is not statistically established. If the flips are not robust to training objective or seed, the first half of the central claim ('model-superiority conclusions ... are conditional') is not empirically demonstrated for the original CTDG benchmarks, even though the theoretical possibility remains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies how the choice of negative-sampling protocol affects MRR-based evaluation of continuous-time dynamic graph (CTDG) link predictors. It derives the Bayes-optimal ranking under a sampled candidate set (Eqs. 2-3), shows that uniform finite-K evaluation introduces an exact expected-reciprocal-rank transformation of the full-catalog rank (Eqs. 6-7), and analyzes a minimal pair-membership scorer (Eq. 10). The empirical section compares six models on LastFM, MOOC, Reddit, and Wikipedia under a unified BCE training recipe, computing expected Uniform-20 MRR and full-catalog MRR from the same score vectors. The paper reports model-order reversals on three datasets, module-effect sign changes for CRAFT-R versus CRAFT, and a 2x2 membership diagnostic. It concludes that sampled-negative model-superiority and ablation conclusions are conditional on the candidate configuration and recommends all-entity ranking as primary evidence for fixed-catalog benchmarks.","tokens_in":1759,"tokens_out":1784,"duration_ms":73789,"significance":"If the empirical claims hold, this is a valuable contribution to CTDG evaluation methodology. The theoretical core is clean and parameter-free: Eq. (7) gives an exact, closed-form relationship between an event's full-catalog rank and its expected MRR under uniform K-negative sampling, and Eq. (10) provides a simple analytic benchmark for the pair-membership signal. The 2x2 diagnostic and the representation projection are useful tools for separating membership-aligned signals from fine-grained pair compatibility. The recommendation to report all-entity MRR alongside sampled protocols is actionable and directly addresses a real source of protocol-dependent conclusions. The paper also ships reproducible code and a public data-materialization pipeline, which strengthens confidence. The main weakness is that the headline empirical demonstration rests on a non-standard training recipe and three seeds; the observed model-order reversals and module-effect flips are not yet shown to be robust across training objectives and seed realizations. The theoretical and mechanistic parts are solid; the empirical part needs additional support or a narrowing of the claim.","major_comments":[{"comment":"All six models are trained with a single non-standard recipe: 3 epochs, BCE, one random negative per positive, and three seeds. This is not the original training configuration for several models (e.g., CRAFT uses BPR in its original study). Table IX shows that the key Wikipedia module-effect sign flip — CRAFT-R minus CRAFT under Expected-20 (+0.0037) versus all-entity (−0.0266) under BCE — becomes +0.0345 and +0.0629 under BPR. Thus the empirical demonstration that sampled versus full-catalog evaluation flips a module conclusion is currently conditional on the BCE training objective. Since the abstract states that the results 'establish' that model-superiority and ablation conclusions are conditional on the candidate configuration, this load-bearing empirical claim needs to be either established under the original training recipes with more seeds, or explicitly scoped to the unified BCE","section":"§V-A, §V-G, Table IX"},{"comment":"The central empirical result — 'at least one model pair changes relative order on three of the four datasets' — is inferred from mean MRR values over three seeds. No seed-paired confidence intervals or permutation tests are reported for the model-pair order differences. For example, on LastFM the reversal between CRAFT and DyGFormer is CRAFT .4283±.0112 vs. DyGFormer .4161±.0015 under Expected-20, and .0636±.0065 vs. .0811±.0100 under the full catalog. With n=3, this could be a seed artifact. Table VIII provides seed-paired intervals only for CRAFT-R minus CRAFT, not for the other model-pair reversals. Since the headline claim is empirical, the paper should provide paired evidence on the distribution of the sign of the model-order differences across seeds, or at least a paired bootstrap over seeds. Without this, the 'one pair flips on three datasets' statement is not statistically establ","section":"§V-E, Table V, Table VI"},{"comment":"The 'all-entity' risk set is fixed to destinations appearing in the training prefix, and test events whose positive destination is not in that set are excluded (catalog coverage 0.987–1.000). For Wikipedia and Reddit, new destinations continue to arrive after the training prefix, so neither the sampled protocol nor the fixed-catalog protocol directly corresponds to an open-world deployment target. The paper acknowledges this in Section I and Section VI, and the formal claim is scoped to 'an exactly enumerable fixed catalog.' However, the abstract and title omit this qualifier, and the general recommendation to use all-entity ranking as primary evidence could be misread as applying to open-world settings. The discrepancy between the fixed-catalog frame and the actual data-generating process should be stated more prominently, particularly because the paper uses Wikipedia and Reddit as two","section":"§IV-C, §VI"}],"minor_comments":[{"comment":"The displayed formula for Eq. (15) in the appendix has a formatting issue: the numerator is written as `{ N \\choose K+1} - {N-r \\choose K+1}/{r {N-1 \\choose K}}`, which is ambiguous and likely missing parentheses. The intended expression should be `[(N \\choose K+1) - (N-r \\choose K+1)] / [r (N-1 \\choose K)]`.","section":"Appendix A, Eq. (15)"},{"comment":"The column header 'History β' is not defined in the table. It should be labeled 'source-history β*' or 'β*_hist' to distinguish it from a coefficient computed from event history in a different sense. Also, the exact equality between analytic and learned β under random sampling is striking and should be stated explicitly as a check of Eq. (10) rather than left implicit.","section":"Table III"},{"comment":"The abstract's 'These results establish ...' is stronger than the evidence supports, given the training-objective sensitivity shown in Table IX. Softening to 'provide evidence' or 'demonstrate under the unified BCE recipe' would better match the content.","section":"Abstract, §V-G"},{"comment":"The label 'All-entity decrease' on the right of the horizontal axis is ambiguous. It would be clearer to mark the full-catalog point for Wikipedia explicitly, since the sign change on Wikipedia is one of the main findings.","section":"Figure 2"},{"comment":"The reference to Krichene and Rendle and Dallmann et al. is appropriate, but the discussion could briefly note that the CTDG setting adds time-dependent pair membership, which is the novel mechanism. This is implied but would help the reader.","section":"§II-B"}],"recommendation":"major_revision","confidential_remarks":"The theoretical and mechanistic parts of the paper are strong and likely correct; the exact expected finite-K MRR formula and the membership diagnostics are useful contributions. My main concern is the empirical support for the headline reversal claim: it is based on a unified BCE recipe and three seeds, and the one training-objective sensitivity test included (Table IX) shows that a central sign flip reverses under BPR. The authors should either run the full model comparison under original training recipes with more seeds, or clearly restrict the claim to the unified recipe. Adding seed-paired statistical tests for the model-pair order reversals is essential. With that revision, the paper would be a solid methodological contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one. It makes a claim that matters for the CTDG benchmark literature: sampled-negative MRR is a joint product of scorer and candidate protocol, so model rankings and module effects are conditional on q and K. The paper earns that claim. Section III derives the Bayes-optimal density-ratio score, the hypergeometric expected-RR identity, and the membership coefficient; all check out. The empirical core is thoughtful: same checkpoints, same score vectors, nested candidate sets, plus a 2x2 membership diagnostic and a representation projection with random/shuffled controls. They also report their own limitations—3-epoch unified BCE training, fixed training-prefix catalog, BPR sensitivity for CRAFT/CRAFT-R. That is more honest than most benchmark papers.\n\nThe central observation holds: on LastFM, CRAFT and DyGFormer reverse between Expected-20 and full catalog, and the CRAFT-R module gain on Wikipedia changes sign as K grows. Even if those specific flips were training-recipe artifacts, the paper's broader point—that within a fixed checkpoint the comparison problem changes with q and K—is demonstrated repeatedly.\n\nThe soft spots are real but not load-bearing. All six models are trained with 3-epoch BCE and one negative per positive, which is not what the original models used. Table IX shows the Wikipedia sign flip reverses under BPR, so the headline examples are configuration-dependent. Three seeds is thin, and the paper does not provide seed-paired intervals for the model-pair order reversals, only for module effects. The stress-test worry about seed robustness is legitimate; I would want to see whether the LastFM flip survives more seeds and original training recipes before treating the specific model-order conclusions as stable facts. The fixed-catalog scoping is stated clearly in Sections I and VI, but the abstract's recommendation is easy to over-read as general; they should foreground that it applies only to enumerable fixed catalogs.\n\nThe citation pattern is fine; they credit Krichene-Rendle, Dallmann, NCE, and the JODIE all-item lineage. No circularity—the theory is parameter-free and the empirical claim is observed.\n\nWho this is for: anyone doing architecture comparisons on CTDG benchmarks, especially those relying on Random-20 MRR as a headline number. It deserves a serious referee. The paper needs a robustness pass (more seeds, original training objectives for all models, seed-paired order CIs) but the core argument is sound.\n\nRecommendation: send to peer review with a request for that robustness pass.","headline":"A careful, honest paper showing sampled-negative MRR in CTDGs is protocol-conditional; the main empirical flips are real but partly configuration-dependent, and the fixed-catalog recommendation is well scoped.","tokens_in":14515,"tokens_out":1939,"would_cite":true,"duration_ms":17217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sampled-negative evaluation can flip which CTDG model is ranked best, so model-superiority conclusions are conditional on the sampler and candidate count.","keywords":["continuous-time dynamic graphs","negative sampling","all-entity ranking","temporal link prediction","MRR evaluation","pair-history membership","model comparison","candidate protocol"],"falsifier":"Re-run the six models on the four datasets with the same checkpoints and scorers and compute both expected Uniform-20 and full-catalog MRR; if no model pair reverses relative order on any dataset and the CRAFT-R module gain keeps the same sign for every K and the full catalog, the paper's central empirical claim fails. A cheaper check: on the released code, verify whether any test event's sampled candidate set can contain a positive that is unseen in the training prefix yet still scored against the fixed catalog, which would expose a risk-set mismatch.","tokens_in":13340,"feed_emoji":"🎯","tokens_out":4110,"duration_ms":39938,"temperature":0.7,"pith_summary":"This paper tries to establish that MRR scores on continuous-time dynamic graph benchmarks depend on the negative-sampling protocol — the distribution q and the number of candidates K — not just on the model being evaluated. It derives the Bayes-optimal score for sampled ranking as the density ratio log p+(v|x) - log q(v|x), so any non-uniform sampler changes the ranking target, while a uniform finite candidate set sees only a nonlinear partial view of the full catalog rank. Empirically, across six models and four datasets, at least one model pair reverses order between expected Uniform-20 and full-catalog MRR on three datasets, and the CRAFT-R module gain on Wikipedia changes from +0.0037 at K=20 to -0.0266 over the full catalog. The mechanism is pair-history membership: models that use source-destination history directly transmit the sampler's seen/unseen composition into the score. The paper recommends all-entity ranking over a fixed catalog as the primary evidence for architecture comparisons.","feed_headline":"Sampled negatives flip CTDG model rankings on three datasets","feed_subtitle":"Uniform-20 and full-catalog MRR disagree on which model is best; all-entity ranking removes the choice.","key_machinery":"The load-bearing objects are three. First, the density-ratio score ℓ*_q(x,v) = log p+(v|x) - log q(v|x): this is the Bayes-optimal ranking for a sampled candidate set, showing that a non-uniform negative distribution changes what optimal ranking means. Second, the hypergeometric expectation E[RR_K | r, N] (Eq. 7): it maps each event's full-catalog rank r into expected finite-K MRR, isolating the pure effect of K on the metric and removing seed variance. Third, the minimal pair-membership scorer β(M) = b + βM, whose optimal coefficient β*_q = logit(p1) - logit(q1) quantifies how much of sampled MRR can be earned just by knowing whether the pair has been seen before. Together these show how pa","core_discovery":"The central claim is that a sampled-negative MRR is a joint product of the scoring function and the test-time candidate protocol, so model-superiority and ablation conclusions are conditional on q and K. The paper derives the Bayes-optimal score for sampled ranking as log p+(v|x) - log q(v|x), shows that a uniform q over a fixed catalog preserves the population-optimal order while finite-K MRR is only a partial view, and proves the expected finite-K MRR is a hypergeometric quantity determined by the positive's full rank. Pair-history membership — whether the source-destination pair has appeared before — can nearly separate positives from negatives in small random candidate sets. With identic","pith_inferences":["If open-world arrivals matter, neither sampled MRR nor training-prefix all-entity MRR is the deployment target; the paper's recommendation implicitly assumes the destination catalog stops growing at the training prefix, so applying it to Wikipedia/Reddit-style streams requires a time-available or expanding-catalog risk set.","The density-ratio view suggests the negative sampler is part of the task definition; one could invert the paper's logic and tune q to define a preferred ranking target, making 'which q' a modeling choice rather than an evaluation nuisance.","The representation-intervention result invites a testable extension: if pair-membership direction is removed from CRAFT-R's pre-scorer, retrained models should show reduced Repeated-Unseen amplitude, and all-entity MRR should become a stronger predictor of downstream ranking quality.","Since the training objective changes the sign of the module effect (BCE vs BPR on Wikipedia), architecture comparisons should use a unified objective and treat the objective as a confound, not just the evaluation protocol."],"forward_implications":["Reported model rankings from random-20 or uniform-20 CTDG benchmarks should be read as statements about that candidate configuration, not about the architectures in general.","Ablation studies that add a pair-history or recurrence module should report full-catalog MRR, because the module's measured gain can change sign with K and training objective.","All-entity ranking over a fixed training catalog removes negative-selection freedom and sampling variance while retaining the original scorer, making architecture comparisons stable and reproducible.","Support-restricted samplers such as source-history sampling can make a new positive untestable by construction, since the positive lies outside the sampler support while negatives lie inside it.","Expected finite-K MRR computed from full ranks via Eq. (7) is a seed-free metric that separates candidate-draw variance from the K-dependent transformation."],"fun_headline_variants":["Sampled negatives flip model rankings in CTDGs","CTDG rankings hinge on negative sampling choice","All-entity ranking removes CTDG benchmark bias","Sampled-negative MRR is protocol-dependent","Negative sampling changes CTDG model order"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole comparison assumes the right risk set is the fixed catalog of destinations seen in the training prefix; if the deployment setting admits new destinations after the prefix, both sampled MRR and the proposed all-entity MRR are measuring a target that is not the test-time ranking.","fun_headline_variants_meta":{"raw":{"variants":["Sampled negatives flip model rankings in CTDGs","CTDG rankings hinge on negative sampling choice","All-entity ranking removes CTDG benchmark bias","Sampled-negative MRR is protocol-dependent","Negative sampling changes CTDG model order"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1130,"prompt_tokens":785,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":529,"tokens_out":345,"duration_ms":3652,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:10:41.908879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the six models on the four datasets with the same checkpoints and scorers and compute both expected Uniform-20 and full-catalog MRR; if no model pair reverses relative order on any dataset and the CRAFT-R module gain keeps the same sign for every K and the full catalog, the paper's central empirical claim fails. A cheaper check: on the released code, verify whether any test event's sampled candidate set can contain a positive that is unseen in the training prefix yet still scored against the fixed catalog, which would expose a risk-set mismatch.","supporting_citations":[],"review_version":1}