REVIEW 4 major objections 4 minor 10 references
Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that per-sample noise levels, set inversely proportional to local density estimated by KDE, improve Discrete Walk-Jump Sampling for antibody sequence generation.
desk verdict A plausible data-dependent noise scaling for walk-jump sampling that is undermined by an unspecified sigma conditioning step; worth reviewing but needs a major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the data-dependent noise scale σ(x), a per-sample smoothing bandwidth set inversely proportional to the local KDE density, giving the method its name DDS-dWJS. It is inserted into the Neural Empirical Bayes denoising objective L(φ) = E[||x − x̂_φ(y)||²] with y = x + N(0, σ(x)²), so the score network learns to denoise with a scale that adapts to local density. KDE over six biochemical features supplies the density estimate; Random Fourier Features make that estimate tractable for 1.3 million sequences; at inference the maximum per-sample σ is used to initialize the walk and for the jump step.
What would settle it
Compute local densities for a held-out set of antibody sequences twice: once with the KDE over the six biochemical features used in this paper, and once with a direct estimate in the full one-hot embedding (for example, mean distance to k nearest one-hot neighbors on a subsample). If the two rankings disagree substantially, the proxy fails. Alternatively, retrain with the fitted per-sample σ values randomly permuted across training samples: if β-sheet content and FID do not degrade, local density is not the causal ingredient.
Extended reading notes
Core claim
The paper claims that no single global noise level σ is right for antibody sequence data because sequences are unevenly sparse: a few dense clusters, most samples in varying sparse regions. It therefore sets per-sample noise σ(x) ∝ 1/p̂(f(x)), where p̂ is a KDE density over six biochemical features, accelerated with random Fourier features. Training the same ByteNet denoiser with these sample-specific σ values and using σ_max for initialization/jump gives consistent improvements in β-sheet content, instability index, and FID over fixed σ baselines. On a controlled 4D toy dataset it recovers held-out true modes and produces fewer false modes than the sharpest fixed-σ baseline.
Load-bearing premise
The method assumes that density estimated from six biochemical features (hydrophobicity, molecular weight, isoelectric point, aromaticity, instability index, β-sheet content) tracks the true local density of the 297×21 one-hot sequence space where the score model actually operates; if feature-space density diverges from one-hot-space density, the per-sample σ assignments are misplaced.
Editorial extensions
If this is right
- A single score model trained with per-sample σ can navigate both dense clusters and sparse isolated regions, avoiding the oversmoothing of large fixed σ and the instability of small fixed σ.
- On antibody sequences, DDS-dWJS improves β-sheet content, instability index, and FID over dWJS with σ ∈ {0.5, 1.0, 2.0}, while keeping edit distance and intra-diversity competitive.
- On the 4D toy dataset, DDS-dWJS recovers most true modes, including held-out ones, and suppresses the growing number of false modes that the sharp σ=0.25 baseline produces.
- The added cost is small: a KDE preprocessing pass before the same dWJS training loop, with no architectural change to the score model.
- Data-dependent σ removes the need to grid-search a global noise level, although the scaling bounds [σ_min, σ_max] still have to be chosen.
Reading between the lines
- If density in the six-feature space tracks one-hot-space density, the same recipe should transfer to other sparse discrete biological sequence families such as T-cell receptors or HLA peptides; the paper does not test this.
- A shuffled-σ control would isolate the mechanism: retrain with the same per-sample σ values randomly permuted across training samples. If quality metrics do not degrade, local density is not what drives the improvement.
- The scaling bounds and the KDE bandwidth/RFF dimension are not reported, so a reproducible version should state them and compare feature-space density ranks against nearest-neighbor density in the full 297×21 one-hot embedding.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DDS-dWJS, an extension of Discrete Walk-Jump Sampling (dWJS) for antibody protein sequence generation. Instead of a single global noise scale σ, the method assigns each training sample a per-sample σ(x) that is inversely proportional to a kernel density estimate of the data around that sample. The density estimate is computed on six biochemical features, and the σ values are then normalized to a fixed range. A score/denoising network is trained with these heteroscedastic noise levels and is used at inference with the same walk-jump procedure as dWJS, using σ_max for initialization and the jump. Experiments on a synthetic 4D toy dataset and on a real antibody dataset (POAS) compare DDS-dWJS to fixed-σ dWJS baselines across metrics such as β-sheet content, instability, edit distance, intra-diversity, FID, and DCS. The paper claims consistent improvements from data-dependent noise scaling.
Significance. If the central claim were fully supported, the work would make a useful contribution to generative modeling of sparse, high-dimensional discrete data: replacing a global smoothing scale with local, data-dependent scales is a plausible and potentially important idea, and the synthetic experiment with held-out modes is a nice way to test mode discovery and spurious generation. The paper also provides a clear empirical motivation that protein sequences exhibit heterogeneous sparsity. However, the current manuscript does not establish the theoretical validity of the training objective under per-sample σ, does not validate the density proxy used to set σ, and its headline claim of “consistent improvements” is stronger than Table 1 supports. These are load-bearing gaps, not presentation issues.
major comments (4)
- [Section 2, Eqs. (2)–(3); Section 3.2] The NEB derivation in Section 2 assumes a single fixed noise level: y = x + N(0, σ²I), with the score relation x̂(y) = y + σ² g_φ(y). The paper states in Section 3.2 that the score model is trained “in the same manner” as dWJS but with sample-specific σ values, and it never states that the network receives σ(x) as an input. With heteroscedastic noise y = x + N(0, σ(x)²I), the optimal denoiser minimizing Eq. (3) is the posterior mean E[x|y] under the joint model over (x, σ(x)); this is not equal to y + σ_max² g_φ(y) in general. Thus the walk-jump update using σ_max is not justified as a Neural Empirical Bayes update unless the model is explicitly σ-conditioned or a heteroscedastic extension is derived. This is the central theoretical link between the proposed σ(x) and the reported gains, and it is currently missing.
- [Section 3, Eq. (5); Figure 2] The per-sample σ is defined from a KDE over six biochemical features, while sampling is performed in the 297×21 concatenated one-hot embedding. The paper does not justify or validate that density in this six-dimensional feature space is a reliable proxy for density in the one-hot space where the score network operates. If the proxy ranking is not preserved, the assigned σ values are effectively arbitrary with respect to the sampling geometry. Moreover, the KDE bandwidth h and the number of Random Fourier Features are never reported, so the σ estimates are not reproducible. A validation experiment (e.g., comparing σ rankings from the feature-space KDE against a one-hot-space KDE on a subset) or an ablation would be needed to support Eq. (5).
- [Table 1; Section 3.2] The abstract and Section 3.2 claim “consistent improvements” across metrics, but Table 1 shows that dWJS(0.5) beats DDS-dWJS on edit distance, intra-diversity, and DCS, while DDS-dWJS wins on β-sheet, instability, and FID. The FID comparison is also reported as a single number without standard deviation, so its reliability is unclear. The conclusion in Section 4 correctly says “competitive at diversity and edit distance,” but this contradicts the earlier and abstract-level wording. The claims need to be calibrated to the actual metric-by-metric results, with uncertainty estimates for FID.
- [Section 3.2; Section 5] The per-sample σ values are “normalized and scaled to lie within [0.5, 1]” while the fixed-σ baselines include 0.5 and 1.0. Without an ablation over the normalization range, the observed gains on β-sheet and instability could plausibly arise from the particular choice of range rather than from the density-dependence of σ. Section 5 acknowledges that the range is a free choice but does not test its influence. This is needed to attribute the improvement to data-dependent scaling.
minor comments (4)
- [Figure 1] The caption says “(a) small σ = 0.4, (b) large σ = 1.5, (c) potential optimal σ = 0.8, (d) adaptive σ ∈ {0.2, 1.5}” but the text in Section 1 refers to adaptive values in Figure 1(d) and the nearest-neighbor description with 4 neighbors is inconsistent with the KDE-based σ in Section 3. Please reconcile the figure and text.
- [Section 3] Typo: “anitbody” should be “antibody.” Also, the dataset is called “POAS” in Section 3 but never introduced; please provide a reference or description.
- [Section 3, Eq. (4)] The KDE formula has spacing/formatting issues (“N hd”) that should be fixed. More importantly, h is not specified anywhere in the experiments, and the Random Fourier Feature dimension/count is missing; this prevents reproduction.
- [Section 3.1] The synthetic experiment is described as generating “10× seeds” samples, but the number of seeds and the total number of generations per seed are not clearly defined. Please state them explicitly.
Circularity Check
No significant circularity: data-dependent σ is a preprocessing choice, not a fitted prediction, and the evaluation is against external fixed-σ baselines.
full rationale
The paper's central claim is empirical: per-sample noise levels estimated by KDE improve generation relative to fixed-σ dWJS baselines. The σ(x) values are computed as a preprocessing step from the training data (Eq. 4–5) and are then used to train the denoiser; they are not fitted to the reported metrics (β-sheet, instability, FID, etc.). The comparison against fixed-σ dWJS is an external benchmark, not an identity forced by construction. There is no self-citation: the cited NEB (Saremi & Hyvärinen) and dWJS (Frey et al.) are prior work by other authors. The hand-chosen normalization range [0.5,1] is a tunable hyperparameter acknowledged as a limitation (Section 5), not a fitted parameter renamed as a prediction. The paper also openly states the lack of theoretical understanding (Section 5), which is a limitation rather than a circular step. A separate correctness concern—the score network is never stated to receive σ(x) as an input, so the NEB identity for a single σ may not hold—is a substantive methodological gap, but it is not circularity: it does not reduce the claimed improvement to the method's own inputs. The statement that the σ distribution 'closely aligns with the empirical sparsity structure' is tautological given Eq. (5), but it is only illustrative and not load-bearing for the main empirical claim. Thus no circular step is exhibited.
Assumptions & free parameters
free parameters (3)
- KDE bandwidth h
- sigma normalization range =
[0.5, 1]
- Random Fourier feature count / approximation quality
assumptions (3)
- domain assumption KDE in six biochemical features is a valid proxy for sparsity in the one-hot sequence space.
- ad hoc to paper Score training with sample-dependent sigma remains a valid NEB and denoising objective.
- domain assumption RFF approximates KDE well enough to assign useful sigmas.
Cite this review
Pith. "Pith review of Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling." pith.science (2026). https://pith.science/paper/T6I7SU6X
@misc{pith2026250902069,
author = {Pith},
title = {Pith review of: Data-Dependent Smoothing for Protein Discovery with Walk-Jump Sampling},
year = {2026},
howpublished = {\url{https://pith.science/paper/T6I7SU6X}},
note = {Machine review of arXiv:2509.02069}
}
abstract
Diffusion models have emerged as a powerful class of generative models by learning to iteratively reverse the noising process. Their ability to generate high-quality samples has extended beyond high-dimensional image data to other complex domains such as proteins, where data distributions are typically sparse and unevenly spread. Importantly, the sparsity itself is uneven. Empirically, we observed that while a small fraction of samples lie in dense clusters, the majority occupy regions of varying sparsity across the data space. Existing approaches largely ignore this data-dependent variability. In this work, we introduce a Data-Dependent Smoothing Walk-Jump framework that employs kernel density estimation (KDE) as a preprocessing step to estimate the noise scale $\sigma$ for each data point, followed by training a score model with these data-dependent $\sigma$ values. By incorporating local data geometry into the denoising process, our method accounts for the heterogeneous distribution of protein data. Empirical evaluations demonstrate that our approach yields consistent improvements across multiple metrics, highlighting the importance of data-aware sigma prediction for generative modeling in sparse, high-dimensional settings.
Figures
Reference graph
Works this paper leans on
-
[1]
Pattern recognition and machine learning, volume 4
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006
2006
-
[2]
Diffusion models beat gans on image synthesis
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021
2021
-
[3]
Protein Discovery with Discrete Walk-Jump Sampling
Nathan C Frey, Daniel Berenberg, Karina Zadorozhny, Joseph Kleinhenz, Julien Lafrance-Vanasse, Isidro Hotzel, Yan Wu, Stephen Ra, Richard Bonneau, Kyunghyun Cho, et al. Protein discovery with discrete walk-jump sampling. arXiv preprint arXiv:2306.12360, 2023
work page Pith review arXiv 2023
-
[4]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020
2020
-
[5]
Video diffusion models
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633–8646, 2022
2022
-
[6]
Score-based generative modeling of graphs via the system of stochastic differential equations
Jaehyeong Jo, Seul Lee, and Sung Ju Hwang. Score-based generative modeling of graphs via the system of stochastic differential equations. In International conference on machine learning, pages 10362–10383. PMLR, 2022
work page 2022
-
[7]
Random features for large-scale kernel machines
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007
2007
-
[8]
Exploring protein fitness landscapes by directed evolution
Philip A Romero and Frances H Arnold. Exploring protein fitness landscapes by directed evolution. Nature reviews Molecular cell biology, 10(12):866–876, 2009
work page 2009
Show all 10 references
-
[9]
Neural empirical bayes
Saeed Saremi and Aapo Hyvärinen. Neural empirical bayes. Journal of Machine Learning Research, 20(181):1–23, 2019
2019
-
[10]
Sliced score matching: A scalable approach to density and score estimation
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in artificial intelligence, pages 574–584. PMLR, 2020. 6
2020
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.