scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics

Davide D'Ascenzo; Sebastiano Cultrera di Montesano

REVIEW 3 major objections 5 minor 2 references

scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read scDataset claims that quasi-random block-and-fetch sampling can replace true random sampling for on-disk deep learning on hundred-million-cell single-cell datasets, delivering over 200x faster loading without sacrificing classification…

desk verdict Solid engineering result with a well-measured 204x throughput gain; the parity-with-random-sampling claim is under-supported by one-epoch linear probes and needs revision. read the letter →

arxiv 2506.01883 v3 pith:NKMQ2OCI submitted 2025-06-02 cs.LG cs.AIcs.DBq-bio.GNq-bio.QM

Davide D'Ascenzo , Sebastiano Cultrera di Montesano This is my paper

classification cs.LGcs.AIcs.DBq-bio.GNq-bio.QM

keywords single-cellomicsdataloadingdeeplearningquasi-randomsamplingblockbatchedfetchingAnnminibatchdiversity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the I/O bottleneck that makes deep learning on atlas-scale single-cell datasets impractical can be removed without giving up the diversity that stochastic training needs. It introduces a PyTorch data loader that reads data in contiguous blocks and fetches several minibatches' worth of cells at once, then reshuffles them in memory. On Tahoe-100M, a 100-million-cell perturbation atlas, this quasi-random strategy reads directly from standard AnnData files more than two orders of magnitude faster than true per-cell random sampling. The paper further proves that the loss in minibatch diversity, measured as label entropy, has an explicit bound controlled by block size and fetch factor. If correct, the result means researchers can train on hundred-million-cell datasets on commodity hardware without converting data to new formats or loading the full dataset into RAM.

What carries the argument

The central mechanism is the pairing of block sampling with batched fetching: the dataset's index array is shuffled at the block level, then split into fetches of size m times f, each fetch is sorted back into ascending order so the storage backend can coalesce reads, the contiguous data is loaded, and finally the cells are reshuffled in memory and cut into f minibatches. This preserves random access at the block and fetch level while every disk operation is a contiguous read. The theoretical engine is the plug-in entropy estimator's bias expansion, used to bound expected minibatch label entropy as a function of block size b and fetch factor f.

What would settle it

Train a multi-layer network on Tahoe-100M for at least ten epochs with Adam, comparing BlockShuffling (b=16, f=256) against true random sampling, and measure held-out accuracy on plate 14. A statistically significant gap in macro F1, or a clear entropy deficit persisting at larger block sizes, would refute the claim that quasi-random sampling preserves training efficacy for realistic training runs.

Watch

Extended reading notes

Core claim

The central claim is that quasi-random sampling built from block sampling and batched fetching reproduces the training value of true random sampling while turning disk reads into large contiguous operations. The paper shows analytically that the expected entropy of a minibatch, relative to the dataset's label distribution, is sandwiched between a lower bound set by the block size and an upper bound that is exactly the classical plug-in entropy bias of a multinomial sample; batched fetching moves the effective sample size from the number of blocks toward the number of cells. Empirically, with block size 16 and fetch factor 256, the measured plate-label entropy reaches 3.61 bits versus 3.62 bits for true random sampling on Tahoe-100M, and linear classifiers trained in a single epoch with Adam at learning rate 1e-5 match random sampling's macro F1 across cell-line, drug, and mechanism-of-action tasks. The throughput gain is over 200x relative to the AnnLoader baseline while reading directly from AnnData files.

Load-bearing premise

The training-parity claim rests on the evaluation protocol: one epoch of Adam at learning rate 1e-5 on linear classifiers, repeated twice; if that protocol is too weak to expose a minibatch-diversity deficit, the conclusion would not hold for longer training, deeper models, or stricter statistical power.

Editorial extensions

If this is right

Researchers can train on the full Tahoe-100M atlas on a single workstation with 256GB RAM, without converting data out of AnnData, removing a major barrier to large-scale single-cell deep learning.
The loader applies to any indexable data collection with natural clustering, so shuffled training on time-series, geospatial tiles, or patient-grouped medical images can use the same block-plus-fetch trick.
The entropy bounds give a concrete rule for choosing block size and fetch factor: pick b and f so the lower bound stays within a chosen tolerance of the true label entropy.
Because batched fetching alone accelerates pure streaming by 15x, even inference-only workloads on HDF5-backed AnnData files can run faster with scDataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's entropy bounds assume each block is homogeneous in its label; on real atlases where blocks contain mixed labels the lower bound may be conservative, and the benefit of block sampling could be even larger than the theory suggests.
A natural extension, not explored in the paper, is an automated profiler that samples a small fraction of the dataset to recommend (b, f) for a given hardware and storage backend.
The parity result is demonstrated for one-epoch linear training; whether the mechanism preserves accuracy for many-epoch, deep, or foundation-model-scale pretraining remains open, though the entropy matching suggests the mechanism should transfer.
For backends that only support single-index reads, such as HuggingFace Datasets and BioNeMo-SCDL, the fetch factor gives no benefit and the speedups come entirely from block sampling; the paper reports 47x and 25x gains there.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the throughput result is the real story: 204x over AnnLoader on Tahoe-100M at b=1024, f=1024, a clean 15x from batched fetching on streaming reads, and honest multiprocessing scaling. Second, the claim that quasi-random sampling matches true random sampling for training is not established by Section 4.4. That section trains linear classifiers for one epoch with Adam at lr=1e-5 and two seeds; macro F1 of 0.030 for drug and 0.103 for MoA-fine means barely-trained models, so the protocol cannot expose minibatch-diversity deficits. The abstract says 'across model architectures,' but no non-linear model appears anywhere in the evaluation.

What is genuinely new and good: working directly on standard AnnData without format conversion fills a real gap for atlas-scale single-cell training, and the (b, f) parameterization with the entropy sandwich bound is a clean way to reason about the I/O-versus-diversity trade-off. The theory is derived from stated plate-constant assumptions with the standard Paninski expansion, no fitted constants, and the empirical entropies (1.76 at f=1, 3.61 at f=256 against bounds 1.43-3.63) land where the model predicts. The backend comparison is honest, including the negative result that fetch factor does not help HuggingFace or BioNeMo backends because they lack batched indexing. Related work is current and fair.

Soft spots, in proportion. (1) The parity claim needs either a stronger probe or a softer abstract. The entropy evidence covers only the plate label, not the joint distribution of expression vectors inside blocks, so it does not license conclusions about deeper models or longer training. (2) The proof of Corollary 3.3 asserts monotonicity in f for intermediate fetch factors; the sandwich is proven at the extremes only, so the step should be filled in or the statement relaxed. The empirical data is consistent, so this is minor but real. (3) At B=4 (m=64, b=16), the O(B^-2) terms in the bias expansion are not negligible, making Equation 5 an asymptotic approximation rather than a hard bound. Minor.

Overall: the speedup result holds up, the parity claim is plausible but provisional. The paper deserves a serious referee. A systems reviewer gets real value from the evaluation; a theory reviewer should push on the monotonicity gap. My recommendation: send it to peer review, expecting revision — soften the abstract, add a moderately-trained comparison, and prove or qualify the monotonicity claim.

Referee Report

3 major / 5 minor

Summary. The paper introduces scDataset, a PyTorch IterableDataset for efficient on-disk loading of large-scale single-cell omics data. The method combines block sampling (reading contiguous blocks of cells) with batched fetching (prefetching f minibatches and reshuffling in memory) to reduce random disk I/O while preserving minibatch diversity. The authors derive theoretical bounds on expected plate-label entropy as a function of block size and fetch factor (Theorems 3.1–3.2, Corollary 3.3), validate these bounds on Tahoe-100M, report throughput improvements up to 204× over AnnLoader, and evaluate classification performance on four tasks with linear classifiers. The central claims are that scDataset achieves over two orders of magnitude speedup over true random sampling while working directly on AnnData, and that its quasi-random sampling matches the training performance of true random sampling.

Significance. The paper addresses a practical and currently pressing bottleneck: training deep models on atlas-scale single-cell datasets without format conversion or full in-memory loading. The throughput results are substantial and appear carefully measured (warm-up, parameter sweeps, multiple backends), and the entropy analysis offers a useful, parameter-explicit characterization of the diversity–throughput trade-off. The modular callback architecture and support for multiprocessing/distributed training are genuine engineering contributions that could benefit the scverse ecosystem. The theoretical sandwich bound, if fully established, would be a nice addition to the data-loading literature. However, the empirical 'matches performance' claim is currently supported only by a very low-sensitivity training protocol, and the proof of the general corollary has a gap, so the paper's headline conclusions are not yet fully load-bearing.

major comments (3)

[Section 4.4, Figure 5] The parity claim that underpins the abstract ('matches the performance of true random sampling') rests on a protocol that trains linear classifiers for exactly one epoch with Adam at learning rate 1e-5, repeating each condition only twice. The differences between BlockShuffling (b=16, f=256) and Random Sampling are minuscule (e.g., MoA broad 0.270 vs 0.268; identical values for cell line and drug), while the absolute macro F1 scores for drug (0.030) and MoA fine (0.103) show the models are barely trained. Such a protocol lacks the sensitivity to detect small but systematic deficits in minibatch diversity, and two seeds provide no statistical power. To support the abstract's 'matches the performance' and 'model architectures', the authors should either add multiple epochs, a realistic learning rate, additional seeds, and at least one non-linear model, or temper the claim to the specific linear, single-epoch protocol and remove 'model architectures' from the abstract.
[Appendix C.3, Corollary 3.3] The sandwich bound is stated for any fetch factor f ≥ 1, but the proof only treats the f=1 limit (Theorem 3.2) and the f→∞ limit (Theorem 3.1). The intermediate behavior is dispatched with the assertion that increasing f 'monotonically increases the effective sample size from B toward m', which is stated without proof. Since the lower bound in Corollary 3.3 is supposed to hold for all finite f, this unproven monotonicity is a load-bearing gap. Please supply a formal argument (e.g., a coupling or a direct decomposition of the entropy bias), or explicitly label the monotonicity as an assumption rather than a derived result.
[Abstract and Section 4.1] The headline 'more than two orders of magnitude speedup' is reported for the configuration b=1024, f=1024, which Figure 4 shows to be one that severely suppresses minibatch entropy. The configuration actually used in all training experiments (b=16, f=256) is the one relevant to the paper's end-to-end claim; its throughput (and speedup factor) should be reported prominently in the abstract and in Section 4.1, and the 204× figure should be clearly labeled as the extreme-configuration peak rather than the operating point used for training. As written, readers may infer that near-random diversity and >200× speedup are simultaneously achievable.

minor comments (5)

[Abstract] The abstract claims evaluation 'across multiple classification tasks and model architectures', but Section 4.4 uses only linear classifiers; no architectural variation appears anywhere in the paper. Either add non-linear models or revise the abstract to say 'linear classifiers'.
[Section 4.1, Figure 2] The text states that 'the largest tested values (b=1024, f=1024)' yield the 204× speedup, but it would be clearer to also state the absolute samples/sec for that configuration and for the b=16, f=256 configuration used later, so the reader can directly connect throughput to the training-viable setting.
[Section 3.4, Equation (5)] The numerical bounds shown in Equation (5) are given as 1.43 ≤ E[H(C)] ≤ 3.63, but the text near it says the empirical entropy at f=1 is 1.76±0.33. It would help to also state the corresponding observed entropy at f=256 (3.61±0.08) directly next to the upper bound, as done in the prose, so the validation of the bound is visually explicit.
[Appendix E, Table 2] The text says the configuration with block_size=16, fetch_factor=256, num_workers=4 is 'highlighted in bold', but the table as rendered shows no bold formatting. Please add the bold or remove the reference to it.
[Section 3.1 and Algorithm 1] The notation uses both F_i (integer indices) and calligraphic F_i (loaded data) in the same algorithm; this is understandable but easy to confuse in print. Consider renaming the loaded-data variable (e.g., X_i) in the algorithm and pseudocode.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: throughput and entropy claims are direct measurements or derived from stated assumptions, and no load-bearing self-citation chain was found.

full rationale

The paper's central claims are either direct empirical measurements or consequences of explicitly stated modeling assumptions. The entropy bounds in Section 3.4 are derived from the plate-constant assumption and the standard bias expansion for plug-in entropy estimators (Paninski, 2003), with no constants fitted to the target entropy values. The empirical validation in Section 4.3 measures minibatch plate entropy independently and checks that it falls within the derived interval; this is a consistency check, not a circular prediction. The throughput results in Section 4.1 are direct benchmarks against AnnLoader, with the random-sampling baseline corresponding to block size b=1; this is a parameter setting of the same loader rather than a separately fitted quantity. The classification comparison in Section 4.4 uses one-epoch, low-learning-rate linear classifiers, which may be a weak sensitivity probe and an overclaim relative to the abstract's mention of 'model architectures,' but that is a correctness-risk concern, not circularity. No load-bearing self-citations appear in the references, and no 'uniqueness theorem' is imported from the authors' prior work. The derivation chain is self-contained; the weakest part of the paper is the limited sensitivity of the training-evaluation protocol, not a reduction of the conclusions to their inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central diversity claim rests on the plate-constant model and the monotonicity assumption for intermediate fetch factors; neither is empirically verified outside the specific Tahoe-100M plate structure. No invented physical entities or fitted constants are introduced.

free parameters (2)

block size b = 16 (classification), 1-1024 (benchmarks)
Chosen by hand; controls the I/O diversity tradeoff. The classification experiments use b=16, selected after the entropy analysis in Section 4.3 showed near-random diversity.
fetch factor f = 256 (classification), 1-1024 (benchmarks)
Chosen by hand; controls buffer size and diversity amplification. f=256 yields minibatch entropy close to random sampling in the empirical validation.

assumptions (4)

domain assumption Each contiguous block of size b is homogeneous with respect to a categorical metadata label (plate-constant assumption), Section 3.4.
Underlies the entropy bound; holds approximately for plate labels in Tahoe-100M but not for all metadata granularities such as individual drug-condition labels.
domain assumption For f→∞, the buffer contains each label in proportion to p and selecting m cells is equivalent to IID sampling from p, Appendix C.1.
Used to derive Theorem 3.1; relies on the law of large numbers.
standard math The plug-in entropy bias expansion of Paninski (2003) applies to the minibatch label counts, Appendix C.
Standard asymptotic result used for the bias of the plug-in entropy estimator.
ad hoc to paper Increasing f monotonically increases the effective sample size from B to m, Appendix C.3.
This monotonicity is asserted, not proven; it is used to produce the sandwich bound of Corollary 3.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics." pith.science (2026). https://pith.science/paper/NKMQ2OCI

@misc{pith2026250601883,
  author       = {Pith},
  title        = {Pith review of: scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKMQ2OCI}},
  note         = {Machine review of arXiv:2506.01883}
}

read the original abstract

Training deep learning models on single-cell datasets with hundreds of millions of cells requires loading data from disk, as these datasets exceed available memory. While random sampling provides the data diversity needed for effective training, it is prohibitively slow due to the random access pattern overhead, whereas sequential streaming achieves high throughput but introduces biases that degrade model performance. We present scDataset, a PyTorch data loader that enables efficient training from on-disk data with seamless integration across diverse storage formats. Our approach combines block sampling and batched fetching to achieve quasi-random sampling that balances I/O efficiency with minibatch diversity. On Tahoe-100M, a dataset of 100 million cells, scDataset achieves more than two orders of magnitude speedup compared to true random sampling while working directly with AnnData files. We provide theoretical bounds on minibatch diversity and empirically show that scDataset matches the performance of true random sampling across multiple classification tasks and model architectures.

Figures

Figures reproduced from arXiv: 2506.01883 by the authors.

**Figure 1.** scDataset bridges diverse data backends with PyTorch’s DataLoader through a modular interface. Data retrieval is managed by a configurable fetch callback, followed by preprocessing with fetch transform (e.g., sparse-to-dense conversion). Batches are selected using batch callback and further processed with batch transform before being yielded to the training pipeline. randomly sample from these fixed-size windows. As… view at source ↗

**Figure 2.** Data loading throughput on AnnData as a function of block size and fetch factor. Throughput (samples/sec) increases with both parameters, reaching 204× speedup over AnnLoader at the largest values. AnnLoader Streaming scDataset fetch_factor=1 scDataset fetch_factor=4 scDataset fetch_factor=16 scDataset fetch_factor=64 scDataset fetch_factor=256 scDataset fetch_factor=1024 0 1000 2000 3000 4000 5000 Throughput (sampl… view at source ↗

**Figure 3.** Effect of fetch factor on streaming throughput from AnnData. Batched fetching amortizes fixed I/O overhead, achieving over 15× speedup at f = 1024 compared to iterative minibatch fetching. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Plate label entropy within minibatches as a function of block size and fetch factor. Higher fetch factors compensate for the diversity loss from larger block sizes. Streaming Streaming (buffer) Block size = 16 Fetch factor = 256 Random Sampling 0.9350 0.9355 0.9360 0.9…

**Figure 5.** Figure 5: Classification performance (mean ± std over 2 runs) across four tasks. BlockShuffling with b = 16, f = 256 matches random sampling (b = 1) performance, while sequential and buffered streaming underperform due to plate-scale heterogeneity. 9 [PITH_FULL_IMAGE:figures/fu…

**Figure 6.** Figure 6: Data loading throughput on HuggingFace Datasets. Throughput scales only with block size; fetch factor has no effect due to the backend’s index access pattern. 1 4 16 64 256 1024 Block Size 256 512 1024 2048 4096 8192 16384 Samples Per Second scDataset Throughput with B…

**Figure 7.** Figure 7: Data loading throughput on BioNeMo-SCDL. Similar to HuggingFace, throughput scales with block size while fetch factor provides no additional benefit. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references

[2024]

doi: 10.1093/nar/gkae1142

ISSN 1362-4962. doi: 10.1093/nar/gkae1142. Gayoso, A., Lopez, R., Xing, G., Boyeau, P., Valiol- lah Pour Amiri, V ., Hong, J., Wu, K., Jayasuriya, M., Mehlman, E., Langevin, M., Liu, Y ., Samaran, J., Mis- rachi, G., Nazaret, A., Clivio, O., Xu, C., Ashuach, T., Gabitto, M., Lotfollahi, M., Svensson, V ., da Veiga Bel- trame, E., Kleshchevnikov, V ., Tala...

arXiv
[2025]

Keskar, N

doi: 10.1038/s41467-025-58699-1. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. InInter- national Conference on Learning Representations, 2017. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. InInternational Conference on Lea...

doi:10.1038/s41467-025-58699-1

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [2024]

doi: 10.1093/nar/gkae1142

ISSN 1362-4962. doi: 10.1093/nar/gkae1142. Gayoso, A., Lopez, R., Xing, G., Boyeau, P., Valiol- lah Pour Amiri, V ., Hong, J., Wu, K., Jayasuriya, M., Mehlman, E., Langevin, M., Liu, Y ., Samaran, J., Mis- rachi, G., Nazaret, A., Clivio, O., Xu, C., Ashuach, T., Gabitto, M., Lotfollahi, M., Svensson, V ., da Veiga Bel- trame, E., Kleshchevnikov, V ., Tala...

arXiv

[2] [2025]

Keskar, N

doi: 10.1038/s41467-025-58699-1. Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. InInter- national Conference on Learning Representations, 2017. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. InInternational Conference on Lea...

doi:10.1038/s41467-025-58699-1