Pith. sign in

REVIEW 4 major objections 5 minor 5 references

Power-law scaling in single-cell gene-expression transformers emerges only in data-rich regimes, and the asymptotic floor corresponds to about 2.3 bits of entropy per masked gene.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:53 UTC pith:E5S7KXFA

load-bearing objection Honest first measurement of scaling laws for single-cell transformers, but the data-limited plateau is confounded—the Regime A power law is the solid part. the 4 major comments →

arxiv 2602.15253 v2 pith:E5S7KXFA submitted 2026-02-16 cs.LG q-bio.GN

Scaling Laws for Masked-Reconstruction Transformers on Single-Cell Transcriptomics

Ihor Kendiukhov This is my paper
classification cs.LG q-bio.GN
keywords scaling lawssingle-cell RNA sequencingmasked reconstructiontransformerpower lawdata-to-parameter ratioentropy estimationfoundation models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether the power-law scaling behavior documented for language and vision models carries over to masked-reconstruction transformers trained on single-cell RNA sequencing data. By fitting L = aP^(-α) + c to validation mean squared error across seven model sizes, it finds clear scaling in a data-rich regime (200,000 cells, 512 genes) with α ≈ 0.27 and an irreducible floor c ≈ 1.44, but essentially none in a data-limited regime (10,000 cells, 1,024 genes) with α ≈ 0.009. The contrast is interpreted as evidence that data scarcity, not model capacity, is the binding constraint, and the floor is converted to a preliminary estimate of about 2.30 bits of entropy per masked gene position. A sympathetic reader cares because this gives the single-cell foundation-model community a quantitative way to decide whether adding parameters or adding cells is the better investment, and it provides a first benchmark for how predictable gene expression is from co-expression context.

Core claim

In its own terms, the central discovery is that the validation loss of a masked-reconstruction transformer on single-cell transcriptomes obeys a power-law scaling law with an irreducible loss floor in a data-rich setting, and that the same architecture stops scaling once data are scarce. Specifically, in the data-rich regime the fitted exponent is α ≈ 0.27 (R² = 0.86) with floor c ≈ 1.44 MSE, while in the data-limited regime α ≈ 0.009 with R² = 0.017, meaning parameter count explains almost none of the variance. The authors derive from the floor an entropy estimate of approximately 2.30 bits per masked gene position, with the explicit caveat that this is preliminary because the Gaussian NLL

What carries the argument

The load-bearing object is the masked-reconstruction transformer: a permutation-invariant encoder that treats a cell's gene expression values as an unordered set, masks 15% of genes, and reconstructs them from the rest using a linear per-gene head, trained with mean squared error. The scaling analysis fits the parametric law L(P) = aP^(-α) + c to the best validation MSE at each parameter count; the exponent α is the scaling rate and c is the irreducible floor. The study's interpretive axis is the data-to-parameter ratio, operationalized as the ratio of cells to genes (D/V), which separates the data-rich regime (≈390 cells per gene) from the data-limited one (≈10).

Load-bearing premise

The claim that the Regime B plateau is caused by data scarcity assumes that the two regimes differ only in data-to-parameter ratio, but they also differ in gene vocabulary (512 vs 1,024) and in training duration, with some runs trained for as few as 400 steps; if the larger vocabulary makes reconstruction harder or if unconverged runs flatten the curve, the plateau might not be about data at all.

What would settle it

Train the same architecture at V=512 with D=10,000 cells, running each model to convergence with identical hyperparameters and reporting best validation MSE; if this shows clear power-law scaling, then the Regime B plateau is driven by vocabulary size or training noise, not by data scarcity. Alternatively, measure the fitted exponent α for a fixed V=512 at several D values (10k, 50k, 200k) and check whether α increases monotonically with D.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On a roughly 200,000-cell, 512-gene dataset, model sizes beyond about 20 million parameters offer no further loss reduction; the fitted curve is flat by that point.
  • When the training set is small relative to the gene vocabulary, scaling up the model does not improve validation MSE, so data collection is the priority.
  • The fitted floor c ≈ 1.44 MSE (or ~2.3 bits per masked position) gives a concrete reference value for judging whether a new model or preprocessing pipeline is approaching the limit of predictability.
  • If the scaling law holds, small-scale runs can be used to extrapolate the eventual loss of larger models on the same data regime, informing resource allocation before expensive training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the data-scarcity interpretation would be to keep the gene vocabulary fixed at 512 and vary only the number of cells (e.g., 10k, 50k, 200k); if the scaling exponent grows with cell count, the plateau in the paper's Regime B is truly data-limited.
  • The two entropy derivations agree because they both start from the same MSE floor; a model trained from the start to predict a per-gene variance (Gaussian NLL) would give an independent estimate. If that floor differs materially, the homoscedastic Gaussian assumption is inadequate.
  • The 2.3-bit figure bundles biological noise, technical noise, and information discarded by selecting only highly variable genes; comparing floors across vocabulary sizes or normalization schemes could separate these contributions.
  • If the data-limited plateau replicates, many current single-cell foundation models trained on modest atlases are likely over-parameterized, and compute-optimal training would shift emphasis to much larger curated datasets rather than larger architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a first systematic study of neural scaling laws for masked-reconstruction transformers trained on single-cell RNA-seq data. Using CELLxGENE Census data, the authors construct two regimes: a data-rich regime (V=512 genes, D=200,000 cells) and a data-limited regime (V=1,024 genes, D=10,000 cells). They train seven model sizes from 533 to 3.4e8 parameters and fit the parametric scaling law L = a P^{-α} + c to validation MSE. Regime A yields α≈0.27, c≈1.44, R²=0.86 for the canonical 18-run set; Regime B yields α≈0.009, R²=0.02. The authors interpret this contrast as evidence that data scarcity, not model capacity, is the binding constraint, and they convert the Regime A floor into an entropy estimate of ~2.30 bits per masked gene position. The manuscript includes a reproducibility statement, public code repository, and extensive discussion of limitations.

Significance. If the central comparative claim held, this would be a useful first result for the single-cell foundation-model community: it would establish that parameter scaling of pretraining loss can occur in transcriptomic masked reconstruction, and it would quantify a data-limited plateau that argues for dataset curation over blind model scaling. The paper's strengths include a reproducible setup (public code, deterministic splits, fixed seeds), a clear parametric fit for the data-rich regime, and an unusually candid limitations section that identifies the confounded regime comparison and the derived nature of the Gaussian NLL. These strengths are real but do not by themselves establish the headline claim, because the two-regime comparison changes both V and D simultaneously, and the entropy estimate is not an independent measurement.

major comments (4)
  1. [§3.2, §7.2.2, §10] The central claim that Regime B's flat scaling demonstrates 'data scarcity—not model capacity—is the binding constraint' rests on a confounded design: Regime A (V=512, D=200k) and Regime B (V=1024, D=10k) differ in both vocabulary size and dataset size. The larger vocabulary changes the reconstruction task itself, and Table 2 shows lower absolute MSE in Regime B, so the two tasks are not directly comparable. Section 9 acknowledges this confound, but Sections 7.2.2 and 10 still draw the data-limited conclusion. A matched-vocabulary experiment (e.g., V=512 at D=10k, 50k, 200k) or at least a same-V D-sweep is required before the plateau can be attributed to dataset size rather than task difficulty.
  2. [§7.2.2] The Regime B fit (α=0.009, R²=0.017) is computed from runs with heterogeneous training durations, including runs trained for as few as 400 steps that may not have converged. The manuscript itself says this 'further obscures any underlying scaling trend.' As reported, the flat curve is therefore not a clean estimate of the data-limited scaling behavior; it conflates non-convergence, variable compute, and task differences. The authors should rerun Regime B with standardized steps, batch size, and convergence checks across all model sizes and multiple seeds before using this plateau as evidence.
  3. [§6, Table 3] The canonical Regime A fit uses 18 runs but only six distinct parameter counts; the three seeds per size contribute replication but do not add independent evidence for the trend across P. No confidence intervals are reported for α or c. Given that the entire data-rich scaling claim rests on this fit, bootstrap or profile-likelihood intervals are needed, and a denser sweep in the 10^5–10^7 range (as the authors themselves suggest in §9) would materially strengthen the result. As it stands, R²=0.86 over six parameter values is suggestive but not definitive.
  4. [§7.4, Appendix A] The 2.30-bit entropy estimate is not an independent result. The Gaussian NLL values are derived post hoc from the best MSE values (provenance label 'derived_from_best_mse'), so the agreement between the two entropy derivations is a mathematical consequence of the Gaussian assumption, not independent validation. The Gaussian residual assumption is also unjustified for log(1+x)-normalized expression counts, which are non-Gaussian and heteroscedastic. The paper acknowledges this in §7.4.3, but the abstract and conclusion present the 2.30-bit figure without the needed caveats. Either remove it from the headline claims or label it explicitly as a conditional calculation, not a measurement.
minor comments (5)
  1. [Abstract] The abstract reports 'α = 0.234, R² = 0.823' for the data-rich regime, while Table 3 and §7.2.1 report the canonical α = 0.266, R² = 0.858, and the conclusion uses α≈0.27, R²=0.86. These should be reconciled, and the choice of the canonical 18-run fit should be stated in the abstract.
  2. [Abstract / §4.5] The abstract says parameter counts span 'three orders of magnitude', but the range 533 to 3.4×10^8 spans about 5.8 orders of magnitude. §4.5 correctly says 'approximately five orders of magnitude.' Fix the abstract.
  3. [Abstract] Typo: 'adata-richregime' should be 'a data-rich regime'.
  4. [§5.2 / §7.2.2] Training durations are inconsistent: §5.2 says Regime B uses 20,000–30,000 steps, while §7.2.2 says 20,000–60,000 steps. Clarify which is correct and include per-run step counts in the reproducibility tables.
  5. [Table 1] The XXS parameter count is listed as 534 in Table 1 but the abstract/text says 533. Check the count and ensure consistency.

Circularity Check

1 steps flagged

Entropy 'convergence' is built in from a shared MSE input; the central scaling-law claim is otherwise independent empirical fitting.

specific steps
  1. self definitional [Section 7.4.3 (Caveats) and Appendix A (Eq. 7; Table 4 provenance)]
    "the Gaussian NLL values are currently derived from the MSE values (not independently logged during training), so the two estimates in Table 4 are not fully independent; their agreement primarily reflects the self-consistency of the Gaussian assumption rather than an independent validation. ... all Gaussian NLL values carry the provenance label derived_from_best_mse in the analysis pipeline, confirming that they are not independently measured."

    Equation 7 defines the Gaussian NLL as a deterministic transform of the same MSE values used for the MSE fit: Lhat_NLL = 0.5 ln(2*pi*MSE) + 0.5. Refitting the scaling law on these derived NLL values therefore cannot provide an independent entropy floor; the near-equality of 2.312 bits (from c_MSE) and 2.296 bits (from c_NLL) is a consequence of this transform and the fitting procedure, not evidence about the data-generating process. The paper explicitly concedes this, so the claimed 'convergence' of the two entropy derivations is built in by construction rather than an independent validation.

full rationale

The central scaling-law derivation — fitting L = a*P^-alpha + c to validation MSE across seven model sizes — is empirical and self-contained; it does not reduce to its inputs by construction, and there are no load-bearing self-citations or imported uniqueness theorems. The one genuine circular step is confined to the entropy sub-claim: the Gaussian NLL values are derived from the same best-MSE values (Appendix A labels them derived_from_best_mse), so the agreement between the MSE-based and NLL-based entropy floors is a mathematical consequence of the transform, not independent corroboration. The paper itself acknowledges this in Section 7.4.3. The A/B comparison that supports the 'data scarcity is the binding constraint' conclusion is confounded (V and D change together, and Regime B includes short, possibly unconverged runs), but that is an experimental-validity concern, not a definitional circularity. Score 6 reflects one secondary 'prediction' (the entropy convergence) that reduces by construction, while the main scaling claim retains independent empirical content.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central scaling-law claim rests on three fitted parameters (a, α, c) per regime, a modeling assumption about the functional form, and a set of preprocessing/domain assumptions. The entropy estimate adds a Gaussian-noise assumption and is a transformation of the fitted floor, not an independent measurement. No new physical or mathematical entities are introduced.

free parameters (4)
  • a (scaling constant) = Regime A: 1.750 (all runs) / 2.153 (canonical); Regime B: 1.283
    Multiplicative constant in L=aP^{-α}+c, fitted via OLS on log-transformed data for each regime. Not derived from first principles.
  • α (scaling exponent) = Regime A: 0.234 / 0.266; Regime B: 0.009
    Fitted exponent governing power-law improvement; central to the scaling claim.
  • c (irreducible loss floor) = Regime A: 1.437 / 1.444; Regime B: 0.113
    Fitted via grid search over candidate c in [0, 0.99·min L] to maximize R²; interpreted as irreducible noise/entropy floor.
  • c_NLL (Gaussian NLL floor) = 1.592 nats (Regime A canonical)
    Fitted to the derived Gaussian NLL values, which are deterministic functions of the MSE values; this parameter inherits the MSE fit and is not an independent quantity.
axioms (5)
  • domain assumption The power-law-plus-offset form L=aP^{-α}+c is the correct model for validation MSE vs parameter count in this setting.
    Standard in NLP scaling laws (Kaplan et al., Hoffmann et al.); adopted without domain-specific evidence that it holds for scRNA-seq reconstruction over the studied range.
  • ad hoc to paper Validation MSE with best-checkpoint selection is a faithful proxy for generalization, and the fitted floor c reflects intrinsic data noise rather than under-training or capacity limits.
    The paper interprets c≈1.44 as irreducible noise (Section 7.2.1), but with only six sizes and no compute sweep, the alternative that larger models are under-trained is not excluded.
  • ad hoc to paper Post-hoc Gaussianity: residual errors at masked positions are Gaussian with variance σ²=MSE, justifying the entropy conversion.
    Appendix A derives Gaussian NLL from MSE assuming homoscedastic Gaussian residuals; the paper states this conflates aleatoric and epistemic uncertainty.
  • domain assumption The CELLxGENE Census subset and the Seurat-HVG + log1p preprocessing pipeline are representative enough to support general conclusions about single-cell scaling behavior.
    The 2.30-bit floor is acknowledged to apply only to this specific preprocessing and vocabulary; generalization to other tissues, species, and modalities is untested.
  • domain assumption The Regime A vs Regime B comparison isolates the effect of the data-to-parameter ratio.
    The regimes differ in both vocabulary size and dataset size simultaneously; the paper itself flags this confounding in Sections 3.2 and 9.

pith-pipeline@v1.3.0-alltime-deepseek · 10934 in / 11289 out tokens · 102093 ms · 2026-08-02T22:53:36.978397+00:00 · methodology

0 comments
read the original abstract

Neural scaling laws -- power-law relationships between loss, model size, and data -- have been extensively documented for language and vision transformers, yet their existence in single-cell genomics remains largely unexplored. We present the first systematic study of scaling behaviour for masked-reconstruction transformers trained on single-cell RNA sequencing (scRNA-seq) data. Using expression profiles from the CELLxGENE Census, we construct two experimental regimes: a data-rich regime (512 highly variable genes, 200,000 cells) and a data-limited regime (1,024 genes, 10,000 cells). Across seven model sizes spanning three orders of magnitude in parameter count (533 to 3.4 x 10^8 parameters), we fit the parametric scaling law to validation mean squared error (MSE). The data-rich regime exhibits clear power-law scaling with an irreducible loss floor of c ~ 1.44, while the data-limited regime shows negligible scaling, indicating that model capacity is not the binding constraint when data are scarce. These results establish that scaling laws analogous to those observed in natural language processing do emerge in single-cell transcriptomics when sufficient data are available, and they identify the data-to-parameter ratio as a critical determinant of scaling behaviour. A preliminary conversion of the data-rich asymptotic floor to information-theoretic units yields an estimate of approximately 2.30 bits of entropy per masked gene position. We discuss implications for the design of single-cell foundation models and outline the additional measurements needed to refine this entropy estimate.

Figures

Figures reproduced from arXiv: 2602.15253 by Ihor Kendiukhov.

Figure 1
Figure 1. Figure 1: Validation MSE versus parameter count P on log-log axes, with fitted scaling curves (Equation 3). Each point represents one training run (best checkpoint). Regime A (left) shows clear power-law decay with R2 = 0.82; Regime B (right) is effectively flat with R2 = 0.02. are lower overall (consistent with the larger gene vocabulary providing a richer reconstruction target on normalized data), but the variatio… view at source ↗
Figure 2
Figure 2. Figure 2: Canonical Regime A scaling fits (18 runs, 6 sizes [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 1 canonical work pages

  1. [2016]

    Deep learning scaling is predictable, empiri- cally.arXiv preprint arXiv:1712.00409,

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empiri- cally.arXiv preprint arXiv:1712.00409,

    Pith/arXiv arXiv

  2. [2017]

    Mauck, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satija

    Tim Stuart, Andrew Butler, Paul Hoffman, Christoph Hafemeister, Efthymia Papalexi, William M. Mauck, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satija. Comprehensive integration of single-cell data.Cell, 177(7):1888–1902,

    1902

  3. [2022]

    Gaussian error linear units (GELUs).arXiv preprint arXiv:1606.08415,

    11 Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs).arXiv preprint arXiv:1606.08415,

    Pith/arXiv arXiv

  4. [2023]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    doi: 10.1101/2023.10.30.563174. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. InProceedings of NAACL-HLT,

    work page doi:10.1101/2023.10.30.563174 2023

  5. [2024]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001