{"id":"85457829-ebe8-4c61-bf0e-ba582929e7c1","arxiv_id":"2607.19376","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Refnd builds an HNSW approximate proximity graph and splits by connected components/communities to reduce sequence similarity leakage in ML evaluation, showing lower but more conservative predicted antimicrobial peptide potency performance.","lead":"This paper introduces Refnd, a splitting method that uses an approximate nearest-neighbor graph to keep similar biological sequences out of both training and test sets, reducing information leakage in model evaluation. On an antimicrobial peptide dataset it reports lower but more conservative performance estimates than random or MMseqs2 splits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluated Refnd uses Leiden communities, not the connected components covered by the Sec 3.2 guarantee; the lower PCC is not shown to reflect removal of family-level leakage.","rationale":"The paper's central claim is that RGP justifies component-level splitting, and that the observed performance drop reflects removal of leakage. The most load-bearing link is therefore the identity between the clusters used in the experiments and the latent graph structures in the RGP. The authors are honest that the guarantee is conditional, but the experimental protocol silently substitutes Leiden communities plus post-filtering for the connected components in the formal statement. Post-filtering removes cross-edge training samples, yet it does not prevent two communities from the same latent component being assigned to train and test, so the formal condition in Sec 3.2 need not hold. The max-identity leakage metric cannot detect this failure because it only checks nearest-neighbor identity, not shared family membership. The size-matching and stratified analyses are reasonable controls and support the weaker claim that Refnd restricts train-test similarity, but they do not establish family-disjointness. A direct family-overlap check would settle the issue; until then, the conditional verdict is appropriate.","tokens_in":10981,"tokens_out":6846,"duration_ms":68299,"concrete_test":"On the same 11,059 DBAASP/QMAP peptides, obtain independent family annotations (e.g., Pfam domain families or DBAASP family assignments, or a sensitive cluster set at a much lower identity threshold such as 30% identity). For each of the 10 Refnd community+post-filtering splits, compute the fraction of test peptides whose family also appears in the corresponding training set. If that fraction is non-negligible, the Sec 3.2 guarantee is violated by the evaluated pipeline; if it is zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's guarantee (Eqs 7-9) is explicitly restricted: it holds 'provided the connected components correctly recover the latent graph structures.' The method actually evaluated in Section 5 is the optional variant of Section 3.4: Leiden community detection plus post-filtering (Section 3.5). Post-filtering deletes training samples that share an edge with a test sample, but it does not restore the condition that test clusters are distinct latent graphs. A true RGP component can be internally split by Leiden into communities assigned to different sides of the split; after removing cross-community edges, max-identity between train and test can be below tau while both sides still contain samples from the same latent family. The max-identity analysis (Figure 3, Table 2) therefore does not establish family-disjointness: it only bounds nearest-neighbor identity, a weaker condition. The observed PCC drop (0.599/0.664 to 0.516/0.505) is consistent with the RGP leakage-removal story, but it is also consistent with the extra granularity of communities and the deletion of boundary samples altering the task difficulty. The paper itself flags the unvalidated 'provided' clause but does not supply evidence that components—or the communities actually used—match peptide families.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Relational Generative Process (RGP), a generative model in which datasets arise from latent graph structures (e.g., peptide families), and proposes Refnd, a splitting algorithm that builds a proximity graph via HNSW in O(n log n) time and splits by connected components or, optionally, by Leiden communities with post-filtering. The authors claim that the RGP provides theoretical justification for graph-based splitting, and that Refnd prevents information leakage more effectively than random splitting or MMseqs2 clustering. They validate on an antimicrobial peptide potency regression task, using frozen embeddings and fixed downstream models, and report lower test PCC under Refnd splits, attributing the drop to removal of family-level leakage. The paper includes a max-identity leakage analysis, size matching, and Wilcoxon significance tests across 10 seeds, and releases open-source code.","tokens_in":11230,"tokens_out":2684,"duration_ms":28990,"significance":"If the central claims hold, the paper offers a practically valuable contribution: a scalable, theoretically motivated splitting procedure applicable to proteins, peptides, small molecules, and other relational biochemical data. The experimental design is a strength: fixed embeddings, two predictor architectures, 10 seeds, size matching, and nonparametric significance testing are all appropriate for isolating split effects. The max-identity analysis and stratified performance comparison are useful and clearly presented. The release of the Rust/Python package and experimental code supports reproducibility. However, the core theoretical guarantee—that connected components of the thresholded proximity graph recover the latent graph structures—is asserted conditionally and not validated. Moreover, the evaluated method uses Leiden communities rather than connected components, and the empirical drop in PCC is not conclusively shown to reflect removal of family-level leakage rather than the effect of community granularity and sample deletion. These gaps are load-bearing for the paper's main interpretation, so the manuscript currently warrants major revision.","major_comments":[{"comment":"The central guarantee—'all evaluation samples originate from graph structures g absent from training, provided the connected components correctly recover the latent graph structures'—is conditional on an assumption that is never validated. No evidence is provided that the connected components of the thresholded HNSW proximity graph correspond to peptide families (or to the latent g_i of the RGP). The threshold tau is a free parameter, and the paper explicitly acknowledges this in Sec 3.2 and Sec 6. As stated, the formalism reduces to: if the graph components are the latent families, then splitting by components separates families. This is largely tautological. The authors should validate the recovery condition, for example by using known family/group labels (if available from DBAASP or external clustering) or by constructing synthetic RGP data with known g_i and measuring component purit","section":"Sec 3.2, Eqs (7)–(9)"},{"comment":"The method actually evaluated in Section 5 is the optional Leiden-community variant with post-filtering, not the connected-component split covered by the Sec 3.2 guarantee. As the paper itself notes in Sec 3.4, community detection 'subdivides connected components into smaller, densely connected clusters.' Post-filtering removes training samples that share an edge with a test sample, but it does not ensure that the resulting training and test communities correspond to distinct latent graph structures. A single RGP component can be internally split by Leiden into communities assigned to different sides of the split; after filtering cross-community edges, max-identity between train and test may fall below tau while both sides still contain samples from the same latent family. Therefore the max-identity analysis in Figure 3 and Table 2 bounds nearest-neighbor identity, not family-disjointnes","section":"Sec 3.4–3.5 and Sec 5"},{"comment":"The size-matching protocol controls for the number of training samples, but it does not control for the composition of the training set or the removal of boundary samples. Refnd's post-filtering deletes training samples that are close to test samples; this can preferentially remove easy or informative examples, and can also change the distribution of the test set (since test communities are not balanced by family or identity). The paper attributes the performance drop solely to leakage removal, but no experiment isolates this mechanism. A more direct test would be to compare Refnd against a control where the same graph and communities are used but the split is randomized across communities (without post-filtering), or to evaluate on held-out families identified by an independent clustering and measure transfer performance. As it stands, the conclusion that 'Refnd produces the most faithf","section":"Sec 4, size matching; Sec 5.2, Table 3"}],"minor_comments":[{"comment":"The reference list appears to duplicate the same CD-HIT citation for both [7] and [8], but the text uses [8] for MMseqs2. The MMseqs2 reference (Steinegger and Söding, 2017) is missing. This needs correction.","section":"References [7], [8]"},{"comment":"Typo: 'prosgram' should be 'program'.","section":"Acknowledgments"},{"comment":"The complexity claim of O(n log n) is plausible for HNSW under standard assumptions, but the paper does not report runtime or memory measurements comparing Refnd with O(n^2) baselines. Since scalability is a stated contribution, an empirical runtime plot or table (e.g., wall-clock time vs. n for Refnd, CD-HIT, MMseqs2, and an exact graph construction) would strengthen the paper.","section":"Sec 3.3"},{"comment":"The description 'global protein sequence kernel' at a 50% identity threshold is vague. Please specify the exact kernel formula (e.g., Needleman-Wunsch identity, local alignment identity) and how the threshold tau is mapped to identity. The paper refers to 'identity' but the distance function used in Eq. (6) is not precisely defined beyond the Parasail library.","section":"Sec 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's theoretical framing is a strength if validated, but the current gap between the connected-component guarantee and the evaluated community/post-filtering procedure is substantial. The RGP is essentially a grouped/latent-variable model, and the main theorem-like claim is conditional in a way that makes it close to a reformulation of the splitting objective. That does not make the algorithm useless—there is real value in an O(n log n) approximate method and in the empirical demonstration that split choice affects reported performance—but the paper needs to either validate family recovery on known labels or synthetic data, or substantially soften the theoretical claims. I would support publication after these issues are addressed with additional analyses rather than mere rewriting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the engineering is real: Refnd ships a Rust-backed Python package that builds proximity graphs via HNSW in log-linear time, with modifications to retain edges and low-similarity candidates, plus connected-component, Leiden-community, and post-filtering split modes. For anyone working on peptide or protein benchmark construction, this is a practical contribution. Second, the evaluation is careful: fixed embeddings, size matching across splits, 10 seeds, Wilcoxon tests, and a max-identity analysis that clearly shows Refnd suppresses train/test similarity far better than random or MMseqs2. The drop in PCC (0.599/0.664 to 0.516/0.505) is consistent with removing leakage, and the stratified random-split analysis in Table 2 is a nice piece of evidence.\n\nThe soft spots are real but not fatal. The RGP formalization (Sec 3.1-3.2) is mostly a restatement of the standard hierarchical-generation intuition behind graph-based splitting, and the 'guarantee' in Sec 3.2 is explicitly conditional on connected components recovering latent graph structures. The paper never validates that condition against true family labels. More importantly, the stress-test note is right: the experiments in Sec 5 use the optional Leiden community detection with post-filtering, not the connected components of the theoretical guarantee. Post-filtering deletes cross-boundary training edges, but it does not ensure that each community is a distinct latent family—a single family can be split across communities, so both train and test can contain samples from the same family even after filtering. That means the max-identity analysis only bounds nearest-neighbor identity, not family displacement, and the lower PCC could partly reflect altered task difficulty or finer-grained clusters rather than pure leakage removal.\n\nI don't think this sinks the paper. The authors are transparent about the 'provided' clause, and the method is admittedly heuristic. But the mismatch between the theoretical framing and the evaluated variant should be addressed directly—either by proving something about the community/post-filtering procedure, or by validating against a dataset with known family labels, or by tempering the 'guarantee' language to 'heuristic that reduces leakage.' The complexity claim is also heuristic, since HNSW has no formal O(n log n) worst-case guarantee and the modifications could increase work.\n\nWho this is for: anyone building benchmarks for peptide, protein, small-molecule, or nucleotide ML models. It deserves a serious referee, because it's a well-engineered, reproducible tool with a solid but incomplete empirical story. I'd suggest accepting peer review with a request for a family-label validation or a clearer separation between the theory and the evaluated algorithm.\n\nSerious thinker: yes—the paper is coherent on its own terms and doesn't overclaim in the text, even if the framing is ambitious.","headline":"Refnd is a genuinely useful tool for scalable similarity-based splitting, but the paper's theoretical guarantee covers only connected components, while the evaluated method uses Leiden communities—so the theory and the experiments are somewhat out of phase.","tokens_in":11750,"tokens_out":1050,"would_cite":true,"duration_ms":11886,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Refnd argues that random data splits leak relational structure, and that splitting by inferred family groups gives more honest model evaluation.","keywords":["data leakage","relational generative process","dataset splitting","proximity graph","HNSW","antimicrobial peptides","family-level generalization","sequence identity"],"falsifier":"Run Refnd on a synthetic RGP dataset with known ground-truth families and measure the fraction of test samples whose ground-truth family appears in training. If that fraction is not near zero, the connected-component recovery assumption fails. Alternatively, compare the HNSW-derived components to components of the exact thresholded distance graph: disagreement identifies where the approximation breaks the guarantee.","tokens_in":10806,"feed_emoji":"🧬","tokens_out":3709,"duration_ms":34955,"temperature":0.7,"pith_summary":"The paper argues that standard random splitting overstates model performance on biochemical data because related samples—peptides from the same evolutionary family, compounds from the same scaffold—leak information into the test set. It introduces the Relational Generative Process (RGP), a formal description of datasets as collections of latent graph structures, and derives the split rule that follows: assign whole connected components of a proximity graph to either train or test, so no test sample shares a family with a training sample. The paper then presents Refnd, an algorithm that approximates the proximity graph in O(n log n) time using HNSW and optionally refines components with community detection and post-filtering. On an antimicrobial peptide potency task, Refnd splits lower Pearson correlation from 0.599/0.664 (random) to 0.516/0.505, with the drop attributed to removing leakage rather than to model weakness.","feed_headline":"Refnd splitter drops peptide model score from 0.66 to 0.51","feed_subtitle":"Random train/test splits let related peptides leak; Refnd cuts families apart for more realistic evaluation.","key_machinery":"The load-bearing object is the proximity graph G = (V,E) with edges between samples whose distance falls below threshold τ. Connected components of this graph are treated as inferred RGP graph structures. The construction is accelerated by a modified HNSW (Hierarchical Navigable Small World) build that retains evicted candidates below τ and keeps an explicit edge list, avoiding the O(n²) pairwise matrix. Leiden community detection can subdivide dense components, and post-filtering removes training samples with edges to test samples. The τ threshold is the main interpretable parameter, encoding the expected distance between deployment data and training data.","core_discovery":"The central claim is that relational leakage in biochemical benchmarks is a structural property of the data-generating process, not a modeling artifact. If the data arise from the RGP—sample a latent graph, then sample observations from it—then evaluation is valid only when every test observation's latent graph is absent from training. Refnd operationalizes this: build a proximity graph with an adjusted HNSW build, take connected components (or Leiden communities), assign each cluster wholly to one side, and optionally delete training nodes that still border test nodes. The paper's conditional guarantee is explicit: this excludes all training-present families from evaluation, provided the co","pith_inferences":["A testable extension: on synthetic RGP data with known ground-truth families, Refnd's component recovery rate can be measured directly; leakage should scale with the fraction of missed cross-family edges.","The community-detection variant trades strict family separation for split granularity; the paper's post-filtering mitigates but does not fully eliminate residual leakage, so users needing a hard guarantee should prefer component-based splits.","The τ-tuning procedure recommended in the paper (match test-to-train distances to production-to-train distances) means Refnd's reported performance depends on a deployment assumption; reported numbers may not transfer to different τ or different distance functions.","Neighboring problems could use the same logic: any dataset with latent group structure and a distance satisfying Eq. (5), such as single-cell clones or text from the same underlying generator, might benefit from component-based splitting."],"forward_implications":["On datasets that fit the RGP, evaluation metrics obtained with Refnd are more conservative than random or MMseqs2 splits; differences are statistically significant across 10 seeds (p<0.01, Wilcoxon).","Because HNSW runs in O(n log n), family-disjoint splitting scales to datasets too large for pairwise-similarity clustering.","The drop in MLP advantage under Refnd suggests reported capacity gains can be inflated by leakage; comparisons should control split protocol.","The max-identity distribution gives a direct diagnostic: a split that leaves a thick tail above the family threshold is admitting leakage.","Refnd is applicable beyond peptides to any RGP-type data: small molecules (Tanimoto), structures (TM-score), nucleotide sequences."],"fun_headline_variants":["Refnd splitter prevents peptide leakage, yields honest scores","Relational splits beat random: Refnd reveals true model performance","Refnd: cut relational leakage, get realistic peptide evaluation","New algorithm Refnd splits by graph structure to stop data bleed","Refnd: no more inflated metrics from leaking relational datasets"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the HNSW proximity graph built at threshold τ correctly recovers the true latent graph structures; if bridging edges are missed, τ is mis-set, or the intended deployment includes families already present in training, the leakage-reduction claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Refnd splitter prevents peptide leakage, yields honest scores","Relational splits beat random: Refnd reveals true model performance","Refnd: cut relational leakage, get realistic peptide evaluation","New algorithm Refnd splits by graph structure to stop data bleed","Refnd: no more inflated metrics from leaking relational datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1019,"prompt_tokens":653,"completion_tokens":366,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":397,"completion_tokens_details":{"reasoning_tokens":293}},"tokens_in":397,"tokens_out":366,"duration_ms":4027,"temperature":1.0,"reasoning_tokens":293,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:13:53.518764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Refnd on a synthetic RGP dataset with known ground-truth families and measure the fraction of test samples whose ground-truth family appears in training. If that fraction is not near zero, the connected-component recovery assumption fails. Alternatively, compare the HNSW-derived components to components of the exact thresholded distance graph: disagreement identifies where the approximation breaks the guarantee.","supporting_citations":[],"review_version":1}