{"id":"603e00f8-c208-4477-8aaa-577099285dae","arxiv_id":"2512.18454","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A diffusion model's probability-flow trajectories can rank protein-ligand complexes by typicality, and trajectory statistics improve out-of-distribution detection beyond likelihood alone.","lead":"The paper describes a diffusion-model-based system that scores how typical a protein-ligand complex is compared to the training set, to flag new complexes a model may mispredict. It also shows that adding 18 trajectory-shape features improves OOD separation, though the final classifier needs some labeled out-of-distribution examples, so the label-free claim is overstated.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Trajectory-aware LDR detector uses OOD labels from the target family; the 'label-free' claim is unsupported.","rationale":"The reader's weakest_assumption identified two related issues: Assumption S1.8's fitted monotone envelope and the trajectory-aware LDR's use of OOD samples from the same held-out family. I focus on the latter because it is the more direct threat to the paper's central claim of a label-free OOD detector. The method section is explicit that the OOD KDE and decision threshold are fit on examples from 'a given shift condition,' and the bootstrap procedure partitions that same OOD split into training/validation/test. Thus the reported AUROC numbers in Table 1/S3 are not achievable when a genuinely new family arrives without OOD labels. The baselines, by contrast, are strictly ID-trained, so the comparison is not apples-to-apples. The likelihood-only results are promising and are not undermined by this concern, but the paper itself concedes they fail on 3dd0 and are insufficient alone; the high-sensitivity detector is the label-using component. This supports the reader's REJECT verdict as is, so no verdict adjustment is needed.","tokens_in":37973,"tokens_out":4281,"duration_ms":42081,"concrete_test":"Re-run the trajectory-aware evaluation with the OOD KDE removed: score each test complex solely by the negative log-density of the ID KDE in the same 15-dimensional PC space, with the threshold fixed as a quantile of ID training scores and no OOD labels used anywhere. If the mean AUROC/accuracy over the seven family splits drops substantially relative to Table S3 (e.g., below the GEMS embedding baseline on the same splits), the reported performance depends on transductive access to the target family, and the label-free claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'label-free' claim rests on the trajectory-aware LDR classifier, but Section 4.5 fits a class-conditional OOD KDE phat_OOD to X_OOD = {phi(x_j)}_{j in O}, where O indexes 'OOD calibration samples for a given shift condition.' In the experiments these are complexes from the same held-out protein family that later forms the test set (Section 4.5, 'Bootstrap evaluation under class imbalance'). The decision threshold tau is also chosen on a labeled calibration set of ID and OOD examples from that split. Table 1 and Table S3 therefore measure transductive performance, not unsupervised OOD detection for an unseen family. The abstract's 'label-free OOD quantification workflow' and the claim that the diffusion model 'requires no labels' are contradicted by the method's own construction. This also invalidates head-to-head comparison with the GEMS embedding-space and Rate-In baselines, which are trained on ID data only and never see target-family OOD examples. The likelihood-only results in Sections 2.3 and 2.4 remain unsupervised, but the paper explicitly shows they are insufficient on the 3dd0 family (Section 2.2), so the high-sensitivity trajectory detector is the component that carries the advertised claim; and that component is not label-free.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a diffusion-based OOD detection framework for 3D irregular molecular graphs, specifically protein–ligand complexes. It introduces a unified continuous diffusion over coordinates and categorical embeddings, uses the PF-ODE to compute per-complex log-likelihoods, and augments them with 18 trajectory statistics. A trajectory-aware log-density-ratio (LDR) classifier is then evaluated on held-out protein-family OOD datasets and compared with post-hoc baselines built on the GEMS binding-affinity encoder. The paper also claims that PF-ODE likelihoods correlate with GEMS prediction errors and that a formal theorem guarantees error control with high probability. The likelihood-only analysis is unsupervised; the trajectory-aware LDR detector, however, fits its OOD density on labeled examples from the same target family used for testing, which undermines the paper's central 'label-free' claim.","tokens_in":38341,"tokens_out":3665,"duration_ms":50917,"significance":"If the central claim were sound, this would be a valuable contribution: a single unsupervised generative model acting as a reliability certificate for geometric deep learning predictors would have broad applicability in drug discovery and structural bioinformatics. The paper contains several genuine strengths: the bioinformatic construction of OOD splits by withholding entire protein families, the careful demonstration that likelihood-only scores align with structural similarity measures, the explicit correlation of likelihoods with an independent model's errors, and a detailed trajectory-feature analysis. The likelihood-only portions of Sections 2.2–2.4 are unsupervised and empirically informative. However, the headline trajectory-aware detector is not label-free, and the formal error-control result is conditional on an assumption that is fitted to the same data it is claimed to explain. Because these issues affect the paper's central advertised contribution, the significance of the results as presented is substantially lower than claimed.","major_comments":[{"comment":"The trajectory-aware LDR detector is not label-free, contrary to the abstract and Section 3. In §4.5, the OOD density is fit to X_OOD = {φ(x_j)}_{j∈O}, where O is defined as 'OOD calibration samples for a given shift condition.' In the experiments these are complexes from the same held-out protein family that is later used as the test set; the decision threshold τ is also chosen on a labeled calibration set of ID and OOD examples. The reported AUROC and accuracy in Table 1 and Table S3 therefore measure transductive performance under access to representative OOD samples from the target shift. This invalidates the claim that the framework 'requires no labels' and undermines the head-to-head comparison with the GEMS embedding-space and Rate-In baselines, which are trained on ID data only and never see target-family OOD examples. The likelihood-only analysis in Sections 2.3–2.4 remains unsu","section":"§4.5, Eq. (4.16)–(4.17), Tables 1 and S3"},{"comment":"The formal error-control theorem is a Chebyshev rearrangement of Assumption S1.8, not a demonstration that diffusion log-likelihoods control GEMS errors. Assumption S1.8 postulates the existence of a non-decreasing envelope φ such that e_θ(x) ≤ φ(L(x)) for almost all x. The proof of Prop. 2.1 and Theorem S1.9 simply combines this assumption with Chebyshev concentration of L(x). The monotonicity of the error–likelihood relationship is assumed, not established. Moreover, the paper states that φ is obtained by fitting a monotone upper envelope to the empirical (L(x), e_θ(x)) scatter in Figure S1, and the reported coverage (75.8%) appears to be computed on the same scatter. This does not support the 'a priori reliability estimate' claim; it is a post-hoc calibration, and no out-of-sample validation of the fitted envelope is provided. The theorem is valid as a conditional statement, but it ca","section":"§2.4, Prop. 2.1; §S1.6.3, Assumption S1.8, Theorem S1.9"},{"comment":"The comparison with baselines is structurally unfair for the reported headline numbers. The trajectory-aware LDR classifier is trained with OOD examples from the target family, while the GEMS embedding-space baseline and Rate-In are trained only on ID data and never observe target-family OOD examples. Table 2 thus compares a transductive method with unsupervised baselines. The fact that Traj-LDR outperforms these baselines on most datasets does not establish the paper's claimed unsupervised advantage. A valid comparison for the label-free claim would require fitting the OOD density on ID data only, or on a disjoint set of held-out families, and then testing on an unseen family; this is not what is done in §4.5 and Tables 1/S3.","section":"§2.5–§2.6, Tables 1 and 2"}],"minor_comments":[{"comment":"Typos and minor presentation issues: 'withing' (Section 2.1), 'out baselines' (Section 2.6), 'reduced trough' (Section 4.1), 'T rajectory statistics' (Section 2.5), 'T able 1' (Table caption). The caption of Figure 3 says 'N=10,510 similarity scores' for all boxplots, but dataset sizes differ; this should be clarified. Figure S1 caption says '100 randomly sampled complexes from each datasets' — should be 'dataset.' The text in Section S1.5 reports different sampling-quality numbers (84% unfragmented, JSD 0.010/0.009) than Section 4.2.4 (72% unfragmented, JSD 0.0017/0.004); the discrepancy should be reconciled.","section":"Various"},{"comment":"The 'Importance Score' in Figure 7 is described as a composite of separation strength and stability under resampling, but the exact formula is not given. Since this figure is used to support the claim that trajectory features contribute beyond log-likelihood, the definition should be provided.","section":"§2.5, Figure 7"},{"comment":"The statement 'We first learn a preprocessing map T = Π_m ∘ Z ∘ Q' is clear, but the choice m=15 for PCA is introduced without justification. Since the ablation of m is not reported, the sensitivity of the LDR results to this free parameter is unknown.","section":"§4.5"}],"recommendation":"reject","confidential_remarks":"The reader's report and the stress-test note identify a load-bearing flaw: the trajectory-aware detector uses OOD labels from the target family, so the paper's central 'label-free' claim is not supported by the experiments. This is not a presentation issue; it affects the validity of the headline results and the comparison with baselines. The likelihood-only analysis is a useful contribution and could be the basis of a revised manuscript with appropriately scoped claims, but as submitted the central claim is not established. I concur with the rejection recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces a diffusion-based OOD detector for protein-ligand complexes, with a genuinely novel twist: a unified continuous diffusion over 3D coordinates and discrete residues/atoms via posterior-mean interpolation, plus a rich set of PF-ODE trajectory features. The likelihood-only results, computed with no labels, show that held-out protein families tend to get lower log-likelihoods and that dataset-level median likelihoods track GEMS test error. That part is real, chemically sensible, and worth building on.\n\nThe central problem is that the high-sensitivity \"trajectory-aware LDR\" classifier is not label-free. Section 4.5 fits the OOD KDE to OOD calibration samples from the same held-out family being tested, and tunes the threshold on a labeled ID/OOD validation split. Table 2 then compares this detector against embedding-space and Rate-In baselines that never see OOD examples. That's an apples-to-oranges comparison, and it invalidates the abstract's \"label-free\" claim. The stress-test note is accurate; I don't think it's a misreading.\n\nThe formal error-control theorem has similar structural trouble. Assumption S1.8 simply asserts that a monotone envelope maps NLL to error upper bounds, and in practice phi is fitted to the very same scatter of the calibration set. So the high-probability guarantee is a Chebyshev rearrangement of an assumed monotonicity, not an independent statistical result. It should be described as a heuristic calibration curve with a tail bound, not as a discovered error-control law.\n\nThe likelihood-only parts of the paper remain unsupervised and are the most defensible contribution, but the paper itself says they are insufficient on the 3dd0 family, and the actual claims of superior detection rest on the flawed LDR component.\n\nAll that said, this is not a sloppy or incoherent paper. The diffusion methodology is carefully described, the feature set is interpretable, and the bioinformatic validation is a good idea. The flaws are fixable: reframe the trajectory-aware detector as transductive or semi-supervised, report what happens when OOD calibration comes only from other families (or none), and soften the \"formal guarantee\" language. If those changes are made, this could be a solid contribution to geometric deep learning reliability.\n\nI'd send it to peer review, but the authors need to address the mismatch between claims and protocol before I'd trust the numbers.","headline":"Useful diffusion-based OOD tool for 3D graphs, but the 'label-free' headline relies on a transductive setup that uses target-family OOD labels; fix the framing and the comparison.","tokens_in":38757,"tokens_out":2817,"would_cite":true,"duration_ms":28886,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that PF-ODE log-likelihoods and trajectory features from a single unsupervised diffusion model can identify held-out protein families as out-of-distribution and forecast the errors of an independent binding-affinity predict","keywords":["out-of-distribution detection","diffusion models","probability-flow ODE","geometric deep learning","protein-ligand complexes","trajectory features","uncertainty quantification","label-free OOD detection"],"falsifier":"Train the trajectory-aware LDR detector using OOD calibration complexes from one held-out protein family and test it on a second, structurally distinct held-out family without any re-calibration; if AUROC drops to chance, the method depends on family-specific calibration rather than a general OOD signature.","tokens_in":37897,"feed_emoji":"🧪","tokens_out":8519,"duration_ms":75350,"temperature":0.7,"pith_summary":"The paper aims to establish that a single diffusion model, trained without labels on the 3D coordinates and chemical identities of protein–ligand complexes, can serve as both an out-of-distribution (OOD) detector and an error forecaster for geometric deep learning models. Log-likelihoods computed along the model's probability-flow ODE identify entire protein families held out from training as OOD, and they correlate strongly with the errors of an independently trained binding-affinity predictor, allowing a priori reliability estimates. Because scalar likelihoods still overlap between ID and OOD classes and suffer from complexity bias, the paper augments them with eighteen trajectory statistics—path tortuosity, flow stiffness, vector-field activity, and related descriptors—and shows that modeling their joint distribution yields a high-sensitivity label-free detector. A formal high-probability error bound is derived under a monotone calibration assumption, linking diffusion likelihood to downstream predictor error. If correct, the work provides a general blueprint for certifying the reliability of predictive models on irregular 3D molecular data.","feed_headline":"Diffusion model flags unseen protein families as out-of-distribution","feed_subtitle":"Unsupervised diffusion model forecasts when molecular affinity predictions can be trusted","key_machinery":"The key machinery is a unified continuous diffusion over the joint state of 3D coordinates and L2-normalized embeddings of discrete atom/residue types, with categorical denoising done by posterior-mean interpolation from softmax logits. This yields a single self-consistent probability-flow ODE (a deterministic ordinary differential equation whose marginals match the diffusion's noisy distributions at every time), and integrating the divergence of its drift along the trajectory gives an exact per-complex log-likelihood via the change-of-variables formula. The same integration supplies eighteen trajectory statistics—path tortuosity, vector-field magnitude and spikiness, Lipschitz stiffness, fl","core_discovery":"The central discovery: a diffusion model's probability-flow ODE behaves differently for in-distribution and out-of-distribution molecular complexes. OOD inputs traverse longer, more tortuous, less stable paths through low-density regions of the learned flow. Integrating the ODE's divergence yields an exact log-likelihood that aligns with bioinformatic similarity and with errors of an independent affinity predictor across eight of nine test sets. Scalar likelihoods fail on one low-complexity family, but eighteen additional trajectory statistics, modeled jointly via a density ratio, fix that failure and reach AUROC above 0.9 on the strongest shifts. A formal bound ties likelihood to predictor","pith_inferences":["The paper's label-free OOD claim presupposes access to representative OOD samples from the target shift: the trajectory-aware KDE in Section 4.5 is fit on OOD complexes from the same held-out family being tested. Applying the detector to a genuinely unseen family with no such calibration sample would be a stronger test than the paper runs.","The formal error-control theorem relies on a monotone calibration curve fitted to the same (log-likelihood, error) scatter used to demonstrate the correlation, so its a priori guarantee is weaker than the statement suggests; the guarantee would be more convincing if the curve were fixed once on a separate calibration set and then locked.","A testable extension is whether the trajectory-feature detector can be trained once on a diverse pool of OOD families and then generalize to unencountered families; the paper's per-family KDE calibration may be transferable if the features capture a universal dynamic signature."],"forward_implications":["Deployed geometric models can be paired with one unsupervised diffusion model to flag inputs whose predictions are likely to be unreliable, without needing task-specific labels.","Benchmark performance claims can be audited: a shallow increase in error as inputs become more OOD indicates true generalization, while a steep increase points to memorization of training motifs.","The trajectory-feature detector corrects the complexity-bias failure of likelihood-only methods, as demonstrated on the low-complexity carbonic anhydrase family.","Because the framework only requires a diffusion or flow-matching model of the in-distribution, it transfers to any data modality with continuous diffusion dynamics."],"fun_headline_variants":["Diffusion model flags unseen protein families","Unsupervised diffusion detects OOD complexes","Trajectory stats beat likelihood for OOD","Diffusion flow reveals out-of-distribution molecules","Model trusts molecular predictions with diffusion"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The label-free detection claim presumes that representative out-of-distribution complexes from the target shift are available to fit the OOD density model, and the error-control guarantee presumes a monotone calibration curve fitted to the very scatter it is meant to bound; either assumption failing would undercut the paper's central claims.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion model flags unseen protein families","Unsupervised diffusion detects OOD complexes","Trajectory stats beat likelihood for OOD","Diffusion flow reveals out-of-distribution molecules","Model trusts molecular predictions with diffusion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000123,"raw_usage":{"total_tokens":968,"prompt_tokens":810,"completion_tokens":158,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":93}},"tokens_in":554,"tokens_out":158,"duration_ms":2616,"temperature":1.0,"reasoning_tokens":93,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:58:56.946755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the trajectory-aware LDR detector using OOD calibration complexes from one held-out protein family and test it on a second, structurally distinct held-out family without any re-calibration; if AUROC drops to chance, the method depends on family-specific calibration rather than a general OOD signature.","supporting_citations":[],"review_version":1}