REVIEW 3 major objections 4 minor 1 cited by
Refnd: Preventing Data Leakage in Relational Datasets
T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Refnd argues that random data splits leak relational structure, and that splitting by inferred family groups gives more honest model evaluation.
desk verdict Refnd is a genuinely useful tool for scalable similarity-based splitting, but the paper's theoretical guarantee covers only connected components, while the evaluated method uses Leiden communities—so the theory and the experiments are somewhat out of phase. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the proximity graph G = (V,E) with edges between samples whose distance falls below threshold τ. Connected components of this graph are treated as inferred RGP graph structures. The construction is accelerated by a modified HNSW (Hierarchical Navigable Small World) build that retains evicted candidates below τ and keeps an explicit edge list, avoiding the O(n²) pairwise matrix. Leiden community detection can subdivide dense components, and post-filtering removes training samples with edges to test samples. The τ threshold is the main interpretable parameter, encoding the expected distance between deployment data and training data.
What would settle it
Run Refnd on a synthetic RGP dataset with known ground-truth families and measure the fraction of test samples whose ground-truth family appears in training. If that fraction is not near zero, the connected-component recovery assumption fails. Alternatively, compare the HNSW-derived components to components of the exact thresholded distance graph: disagreement identifies where the approximation breaks the guarantee.
Extended reading notes
Core claim
The central claim is that relational leakage in biochemical benchmarks is a structural property of the data-generating process, not a modeling artifact. If the data arise from the RGP—sample a latent graph, then sample observations from it—then evaluation is valid only when every test observation's latent graph is absent from training. Refnd operationalizes this: build a proximity graph with an adjusted HNSW build, take connected components (or Leiden communities), assign each cluster wholly to one side, and optionally delete training nodes that still border test nodes. The paper's conditional guarantee is explicit: this excludes all training-present families from evaluation, provided the co
Load-bearing premise
The load-bearing premise is that the HNSW proximity graph built at threshold τ correctly recovers the true latent graph structures; if bridging edges are missed, τ is mis-set, or the intended deployment includes families already present in training, the leakage-reduction claim collapses.
Editorial extensions
If this is right
- On datasets that fit the RGP, evaluation metrics obtained with Refnd are more conservative than random or MMseqs2 splits; differences are statistically significant across 10 seeds (p<0.01, Wilcoxon).
- Because HNSW runs in O(n log n), family-disjoint splitting scales to datasets too large for pairwise-similarity clustering.
- The drop in MLP advantage under Refnd suggests reported capacity gains can be inflated by leakage; comparisons should control split protocol.
- The max-identity distribution gives a direct diagnostic: a split that leaves a thick tail above the family threshold is admitting leakage.
- Refnd is applicable beyond peptides to any RGP-type data: small molecules (Tanimoto), structures (TM-score), nucleotide sequences.
Reading between the lines
- A testable extension: on synthetic RGP data with known ground-truth families, Refnd's component recovery rate can be measured directly; leakage should scale with the fraction of missed cross-family edges.
- The community-detection variant trades strict family separation for split granularity; the paper's post-filtering mitigates but does not fully eliminate residual leakage, so users needing a hard guarantee should prefer component-based splits.
- The τ-tuning procedure recommended in the paper (match test-to-train distances to production-to-train distances) means Refnd's reported performance depends on a deployment assumption; reported numbers may not transfer to different τ or different distance functions.
- Neighboring problems could use the same logic: any dataset with latent group structure and a distance satisfying Eq. (5), such as single-cell clones or text from the same underlying generator, might benefit from component-based splitting.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the Relational Generative Process (RGP), a generative model in which datasets arise from latent graph structures (e.g., peptide families), and proposes Refnd, a splitting algorithm that builds a proximity graph via HNSW in O(n log n) time and splits by connected components or, optionally, by Leiden communities with post-filtering. The authors claim that the RGP provides theoretical justification for graph-based splitting, and that Refnd prevents information leakage more effectively than random splitting or MMseqs2 clustering. They validate on an antimicrobial peptide potency regression task, using frozen embeddings and fixed downstream models, and report lower test PCC under Refnd splits, attributing the drop to removal of family-level leakage. The paper includes a max-identity leakage analysis, size matching, and Wilcoxon significance tests across 10 seeds, and releases open-source code.
Significance. If the central claims hold, the paper offers a practically valuable contribution: a scalable, theoretically motivated splitting procedure applicable to proteins, peptides, small molecules, and other relational biochemical data. The experimental design is a strength: fixed embeddings, two predictor architectures, 10 seeds, size matching, and nonparametric significance testing are all appropriate for isolating split effects. The max-identity analysis and stratified performance comparison are useful and clearly presented. The release of the Rust/Python package and experimental code supports reproducibility. However, the core theoretical guarantee—that connected components of the thresholded proximity graph recover the latent graph structures—is asserted conditionally and not validated. Moreover, the evaluated method uses Leiden communities rather than connected components, and the empirical drop in PCC is not conclusively shown to reflect removal of family-level leakage rather than the effect of community granularity and sample deletion. These gaps are load-bearing for the paper's main interpretation, so the manuscript currently warrants major revision.
major comments (3)
- [Sec 3.2, Eqs (7)–(9)] The central guarantee—'all evaluation samples originate from graph structures g absent from training, provided the connected components correctly recover the latent graph structures'—is conditional on an assumption that is never validated. No evidence is provided that the connected components of the thresholded HNSW proximity graph correspond to peptide families (or to the latent g_i of the RGP). The threshold tau is a free parameter, and the paper explicitly acknowledges this in Sec 3.2 and Sec 6. As stated, the formalism reduces to: if the graph components are the latent families, then splitting by components separates families. This is largely tautological. The authors should validate the recovery condition, for example by using known family/group labels (if available from DBAASP or external clustering) or by constructing synthetic RGP data with known g_i and measuring component purit
- [Sec 3.4–3.5 and Sec 5] The method actually evaluated in Section 5 is the optional Leiden-community variant with post-filtering, not the connected-component split covered by the Sec 3.2 guarantee. As the paper itself notes in Sec 3.4, community detection 'subdivides connected components into smaller, densely connected clusters.' Post-filtering removes training samples that share an edge with a test sample, but it does not ensure that the resulting training and test communities correspond to distinct latent graph structures. A single RGP component can be internally split by Leiden into communities assigned to different sides of the split; after filtering cross-community edges, max-identity between train and test may fall below tau while both sides still contain samples from the same latent family. Therefore the max-identity analysis in Figure 3 and Table 2 bounds nearest-neighbor identity, not family-disjointnes
- [Sec 4, size matching; Sec 5.2, Table 3] The size-matching protocol controls for the number of training samples, but it does not control for the composition of the training set or the removal of boundary samples. Refnd's post-filtering deletes training samples that are close to test samples; this can preferentially remove easy or informative examples, and can also change the distribution of the test set (since test communities are not balanced by family or identity). The paper attributes the performance drop solely to leakage removal, but no experiment isolates this mechanism. A more direct test would be to compare Refnd against a control where the same graph and communities are used but the split is randomized across communities (without post-filtering), or to evaluate on held-out families identified by an independent clustering and measure transfer performance. As it stands, the conclusion that 'Refnd produces the most faithf
minor comments (4)
- [References [7], [8]] The reference list appears to duplicate the same CD-HIT citation for both [7] and [8], but the text uses [8] for MMseqs2. The MMseqs2 reference (Steinegger and Söding, 2017) is missing. This needs correction.
- [Acknowledgments] Typo: 'prosgram' should be 'program'.
- [Sec 3.3] The complexity claim of O(n log n) is plausible for HNSW under standard assumptions, but the paper does not report runtime or memory measurements comparing Refnd with O(n^2) baselines. Since scalability is a stated contribution, an empirical runtime plot or table (e.g., wall-clock time vs. n for Refnd, CD-HIT, MMseqs2, and an exact graph construction) would strengthen the paper.
- [Sec 4] The description 'global protein sequence kernel' at a 50% identity threshold is vague. Please specify the exact kernel formula (e.g., Needleman-Wunsch identity, local alignment identity) and how the threshold tau is mapped to identity. The paper refers to 'identity' but the distance function used in Eq. (6) is not precisely defined beyond the Parasail library.
Circularity Check
Partial circularity: the Section 3.2 'guarantee' is conditional on identifying components with latent graphs, and the max-identity leakage evidence is largely enforced by the same 50% identity post-filtering rule; the PCC drop itself remains empirically independent.
-
self definitional
[Section 3.2, after Eq. (9)]
"This guarantees that all evaluation samples originate from graph structures g absent from training, provided the connected components correctly recover the latent graph structures."
Splitting by connected components (Eqs. 7-9) and the premise 'connected components correctly recover the latent graph structures' are the same statement in different words. The RGP equations (1)-(3) do not imply that thresholded HNSW components or Leiden communities equal the latent g_i; that identity is assumed. The conclusion that evaluation samples come from graph structures absent from training is therefore an unpacking of the premise rather than a consequence derived from the RGP. The paper even flags the 'provided' clause in the same sentence, conceding that the guarantee is definitional.
-
fitted input called prediction
[Section 3.5 and Section 5.1 / Figure 3]
"Post-Filtering ... all training samples sharing at least one edge with a test sample are removed. ... In contrast, the Refnd split concentrates the distribution below the threshold, leaving only a thin tail of residual violations."
The proximity graph edge set is defined as pairs with f(x,x') < tau (Eq. 6), and post-filtering deletes every training sample with an edge to a test sample. Thus max-identity between test and surviving train samples is below the 50% identity threshold by construction. Presenting Figure 3 as evidence that Refnd reduces leakage mostly restates the algorithm's own filtering rule; the thin violation tail is the only non-forced part, arising from HNSW approximation. The leakage metric uses the same kernel and threshold as the split, so it is not an independent confirmation of family-level separation.
full rationale
The RGP formalization itself is a conditional model rather than a mechanical derivation: Section 3.2's guarantee reduces to the 'provided' clause that components recover latent graph structures, and that clause is never independently validated. The optional Leiden community step used in the actual experiments can split a latent family across train/test, so the 'more realistic' conclusion is not established; however, this is a correctness risk more than a circularity. The max-identity analysis is partly circular because the split's post-filtering enforces exactly the below-threshold property that Figure 3 presents as a finding. The central empirical result — lower PCC under Refnd with fixed models and size matching — has independent content and is not forced by construction. Self-citation to QMAP [15] provides the dataset and a threshold recommendation but is not load-bearing for the main derivation. Overall, the circularity is partial: some validation evidence is by construction, while the headline performance comparison remains an independent observation. Score 6 reflects partial circularity, not full equivalence.
Assumptions & free parameters
free parameters (3)
- Proximity threshold tau =
50% sequence identity in experiments (chosen via QMAP production-distribution methodology)
- HNSW construction parameters =
m=16, m_max=16, m_max0=32, m_l=0.36, ef_init=1, ef_construction=128
- Leiden community parameters =
gamma=1.0, beta=0.01, n_iterations=10
assumptions (5)
- domain assumption The observed dataset is generated by sampling m i.i.d. latent graphs g_i and then N_i samples per graph (RGP, Eqs 1-3).
- domain assumption Expected distance between nearby samples in a graph is much smaller than between arbitrary samples (Eq 5).
- ad hoc to paper Connected components of the thresholded proximity graph correctly recover latent graph structures (Sec 3.2, before Eq 7).
- domain assumption The HNSW approximation preserves the connected components that matter (Sec 3.3).
- domain assumption Real-world deployment requires generalization to families absent from training (Sec 3.1, Sec 4).
invented entities (1)
-
Latent graph structure g (per-family generative graph)
Cite this review
Pith. "Pith review of Refnd: Preventing Data Leakage in Relational Datasets." pith.science (2026). https://pith.science/paper/25OCLWZQ
@misc{pith2026260719376,
author = {Pith},
title = {Pith review of: Refnd: Preventing Data Leakage in Relational Datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/25OCLWZQ}},
note = {Machine review of arXiv:2607.19376}
}
read the original abstract
Machine learning models trained on biochemical data are routinely evaluated using splits that fail to account for relational structure, causing information leakage and over-optimistic performance estimates. Existing splitting methods lack theoretical grounding and scale at best quadratically. We introduce the Relational Generative Process (RGP), a mathematical formalization explaining why relational structure arises in biochemical datasets, and Refnd, a splitting algorithm that leverages a proximity graph computed in loglinear time using Hierarchical Navigable Small World (HNSW). We validate on an antimicrobial peptide dataset, showing that Refnd splits yield lower but more realistic evaluation performance than traditional splits. Refnd is applicable to any dataset arising from an RGP such as protein sequences and structures, small molecules, and nucleotide sequences, and is openly available as a Rust accelerated Python package: pip install refnd.
Figures
Forward citations
Cited by 1 Pith paper
-
A Symmetric Layer-Union Audit of Component Collapse in Hierarchical Procedural Corpora
In a fixed six-corpus panel, MyFixit and Doc2Dial both pass the individual content and container layer checks yet fail when the layers are unioned, and the panel cannot separate bridge-specific causes from simple grap...
Reference graph
Works this paper leans on
-
[1]
Do ImageNet Classifiers Generalize to ImageNet?,
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do ImageNet Classifiers Generalize to ImageNet?,” in Proceedings of the 36th International Conference on Machine Learning , PMLR, May 2019, pp. 5389–5400. Accessed: June 15, 2026. [Online]. Available: https:// proceedings.mlr.press/v97/recht19a.html
2019
-
[2]
Hastie, R
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2nd ed. Springer, 2009. [Online]. Available: https://hastie.su. domains/ElemStatLearn/
2009
-
[3]
Cracking the black box of deep sequence-based protein–protein interaction prediction,
J. Bernett, D. B. Blumenthal, and M. List, “Cracking the black box of deep sequence-based protein–protein interaction prediction,” Briefings in Bioinformatics, vol. 25, no. 2, p. bbae76, Mar. 2024, doi: 10.1093/bib/bbae076
-
[4]
Leakage in data mining: Formulation, detection, and avoidance,
S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance,” ACM Transactions on Knowledge Discovery from Data (TKDD) , vol. 6, no. 4, pp. 15:1–15:21, Dec. 2012, doi: 10.1145/2382577.2382579
arXiv 2012
-
[5]
Leakage and the reproducibility crisis in machine- learning-based science,
S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine- learning-based science,” Patterns, vol. 4, no. 9, p. 100804, Sept. 2023, doi: 10.1016/ j.patter.2023.100804
arXiv 2023
-
[6]
REFORMS: Consensus-based Recommendations for Machine-learning- based Science,
S. Kapoor et al. , “REFORMS: Consensus-based Recommendations for Machine-learning- based Science,” Science Advances, vol. 10, no. 18, p. eadk3452, May 2024, doi: 10.1126/ sciadv.adk3452
2024
-
[8]
Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences,
W. Li and A. Godzik, “Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences,” Bioinformatics, vol. 22, no. 13, pp. 1658–1659, July 2006, doi: 10.1093/bioinformatics/btl158
-
[9]
Effect of dataset partitioning strategies for evaluating out-of-distribution generalisation for predictive models in biochem - istry
R. Fernandez-Diaz, H. T. Lam, V. Lopez, and D. C. Shields, “Effect of dataset partitioning strategies for evaluating out-of-distribution generalisation for predictive models in biochem - istry.” Accessed: May 29, 2026. [Online]. Available: https://www.biorxiv.org/content/10.1101/ 2024.03.14.584508v1
2026
Show all 30 references
-
[10]
Mechanisms of protein evolution,
V. Jayaraman, S. Toledo-Patiño, L. Noda-García, and P. Laurino, “Mechanisms of protein evolution,” Protein Science, vol. 31, no. 7, p. e4362, 2022, doi: 10.1002/pro.4362
2022 doi
-
[11]
An efficient algorithm for large-scale detection of protein families,
A. J. Enright, S. Van Dongen, and C. A. Ouzounis, “An efficient algorithm for large-scale detection of protein families,” Nucleic Acids Research, vol. 30, no. 7, pp. 1575–1584, Apr. 2002, doi: 10.1093/nar/30.7.1575
2002 doi
-
[12]
The Properties of Known Drugs. 1. Molecular Frameworks,
G. W. Bemis and M. A. Murcko, “The Properties of Known Drugs. 1. Molecular Frameworks,” Journal of Medicinal Chemistry , vol. 39, no. 15, pp. 2887–2893, Jan. 1996, doi: 10.1021/ jm9602928
1996
-
[13]
Matched Molecular Pair Analysis in Drug Discovery: Methods and Recent Applications,
Z. Yang, S. Shi, L. Fu, A. Lu, T. Hou, and D. Cao, “Matched Molecular Pair Analysis in Drug Discovery: Methods and Recent Applications,” Journal of Medicinal Chemistry , vol. 66, no. 7, pp. 4361–4377, Apr. 2023, doi: 10.1021/acs.jmedchem.2c01787
2023 doi
-
[14]
MoleculeNet: a benchmark for molecular machine learning,
Z. Wu et al., “MoleculeNet: a benchmark for molecular machine learning,” Chemical Science, vol. 9, no. 2, pp. 513–530, Jan. 2018, doi: 10.1039/C7SC02664A
2018 doi
-
[15]
QMAP: A Benchmark for Standardized Evaluation of Antimicrobial Peptide MIC and Hemolytic Activity Regression,
A. Lavertu, J. Corbeil, and P. Germain, “QMAP: A Benchmark for Standardized Evaluation of Antimicrobial Peptide MIC and Hemolytic Activity Regression,” bioRxiv, 2026, doi: 10.64898/2026.02.03.703041. 10
2026 doi
-
[16]
DBAASP v3: database of antimicrobial/cytotoxic activity and structure of peptides as a resource for development of new therapeutics,
M. Pirtskhalava et al., “DBAASP v3: database of antimicrobial/cytotoxic activity and structure of peptides as a resource for development of new therapeutics,” Nucleic Acids Research, vol. 49, no. D1, pp. D288–D297, 2021, doi: 10.1093/nar/gkaa991
2021 doi
-
[17]
Data splitting to avoid information leakage with DataSAIL,
R. Joeres, D. B. Blumenthal, and O. V. Kalinina, “Data splitting to avoid information leakage with DataSAIL,” Nature Communications, vol. 16, no. 1, p. 3337, Apr. 2025, doi: 10.1038/ s41467-025-58606-8
2025
-
[18]
Lo-Hi: Practical ML Drug Discovery Benchmark,
S. Steshin, “Lo-Hi: Practical ML Drug Discovery Benchmark,” Nov. 2023. Accessed: May 29,
2023
-
[19]
GraphPart: homology partitioning for biological sequence analysis,
F. Teufel, M. H. Gíslason, J. J. Almagro Armenteros, A. R. Johansen, O. Winther, and H. Nielsen, “GraphPart: homology partitioning for biological sequence analysis,” NAR Genomics and Bioinformatics, vol. 5, no. 4, p. lqad88, Dec. 2023, doi: 10.1093/nargab/lqad088
2023 doi
-
[20]
AutoPeptideML: a study on how to build more trustworthy peptide bioactivity predictors,
R. Fernández-Díaz, R. Cossio-Pérez, C. Agoni, H. T. Lam, V. Lopez, and D. C. Shields, “AutoPeptideML: a study on how to build more trustworthy peptide bioactivity predictors,” Bioinformatics, vol. 40, no. 9, p. btae555, Sept. 2024, doi: 10.1093/bioinformatics/btae555
2024 doi
-
[21]
SpanSeq: similarity-based sequence data splitting method for improved development and assessment of deep learning projects,
A. Ferrer Florensa, J. J. Almagro Armenteros, H. Nielsen, F. M. Aarestrup, and P. T. L. C. Clausen, “SpanSeq: similarity-based sequence data splitting method for improved development and assessment of deep learning projects,” NAR Genomics and Bioinformatics , vol. 6, no. 3, p....
2024 doi
-
[22]
A density-based algorithm for discovering clusters in large spatial databases with noise,
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, in KDD'96. Portland, Oregon: AAAI Press, Aug...
1996
-
[23]
From Louvain to Leiden: guaranteeing well- connected communities,
V. A. Traag, L. Waltman, and N. J. van Eck, “From Louvain to Leiden: guaranteeing well- connected communities,” Scientific Reports, vol. 9, no. 1, p. 5233, Mar. 2019, doi: 10.1038/ s41598-019-41695-z
2019
-
[24]
Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs,
Y. A. Malkov and D. A. Yashunin, “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 42, no. 4, pp. 824–836, Apr. 2020, doi: 10.1109/ TPAMI.2018.2889473
2020
-
[25]
Community Structure in Graphs,
S. Fortunato and C. Castellano, “Community Structure in Graphs,” Computational Complexity. Springer, New York, NY, pp. 490–512, 2012. doi: 10.1007/978-1-4614-1800-9_33
2012 doi
-
[26]
Parasail: SIMD C library for global, semi-global, and local pairwise sequence align- ments,
J. Daily, “Parasail: SIMD C library for global, semi-global, and local pairwise sequence align- ments,” BMC Bioinformatics, vol. 17, no. 1, p. 81, Feb. 2016, doi: 10.1186/s12859-016-0930-z
2016 doi
-
[27]
US-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes,
C. Zhang, M. Shine, A. M. Pyle, and Y. Zhang, “US-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes,” Nature Methods, vol. 19, no. 9, pp. 1109–1115, Sept. 2022, doi: 10.1038/s41592-022-01585-1
2022 doi
-
[28]
Evolutionary-scale prediction of atomic-level protein structure with a language model,
Z. Lin et al., “Evolutionary-scale prediction of atomic-level protein structure with a language model,” Science, vol. 379, no. 6637, pp. 1123–1130, 2023, doi: 10.1126/science.ade2574
2023 doi
-
[29]
Identification of common molecular subsequences,
T. F. Smith and M. S. Waterman, “Identification of common molecular subsequences,” Journal of Molecular Biology , vol. 147, no. 1, pp. 195–197, Mar. 1981, doi: 10.1016/0022-2836(81)90087-5
1981 doi
-
[30]
A general method applicable to the search for similarities in the amino acid sequence of two proteins,
S. B. Needleman and C. D. Wunsch, “A general method applicable to the search for similarities in the amino acid sequence of two proteins,” Journal of Molecular Biology, vol. 48, no. 3, pp. 443–453, Mar. 1970, doi: 10.1016/0022-2836(70)90057-4. 11 A Supplementary Material A.1 A...
1970 doi
-
[2026]
Available: https://openreview.net/forum?id=H2Yb28qGLV
[Online]. Available: https://openreview.net/forum?id=H2Yb28qGLV
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.