{"id":"bba9e66e-7540-4d38-922a-14d73a0960ed","arxiv_id":"2608.02961","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An RQ-VAE-tokenized causal transformer trained on single-cell gene expression follows a joint two-exponent scaling law in model size and data, with generation quality tracking the pretraining loss.","lead":"The paper trains an autoregressive transformer on single-cell gene expression vectors that are first converted into discrete codes by a learned quantized VAE, and shows the model can generate new expression vectors for a cell type almost as faithfully as real biological replicates. It also fits a two-exponent scaling law linking model size and training data to the pretraining loss, and derives a compute-optimal balance that tilts toward more data at these scales.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The held-out claim in Eq. 2 requires that the RQ-VAE tokenizer never trained on the validation cells; Appendix A.1 trains it on the full scBaseCount atlas, so the validation split may be partly in-sample and the exponents biased.","rationale":"I read the paper as making a bounded but specific claim: for this RQ-VAE-tokenized causal transformer, on this corpus, the pretraining cross-entropy follows Eq. 2 over the tested N×D grid, with positive exponents, and the compute-optimal frontier tilts toward data. The internal evidence for this claim is unusually strong for a small-scale study: a controlled shape-preserving N axis, a proportional-sampling D axis, a 22-point grid, a multi-start Approach-3 fit, residual diagnostics, and four held-out runs predicted within 0.5%. None of my reading found an internal inconsistency in the fitting procedure itself. The one assumption that must hold for the 'held-out' framing is that the validation cells are unseen by every learned component. Appendix A.1 trains the tokenizer on the scBaseCount atlas and freezes it; Appendix B defines the validation split for transformer runs. The text does not state that tokenizer training excluded validation cells. If it did not, L(N,D) is a loss on tokens produced by a component that was fit to those exact validation cells, so the validation split is not fully held out, and the fitted exponents and frontier could be biased. This is not a disagreement with the single-cell scaling debate; it is a data-hygiene check on the paper's own central measure. A code check and, if needed, a tokenizer re-training experiment would settle it. I therefore keep the reader's CONDITIONAL verdict: ACCEPT or REJECT is premature until the split is clarified, but the concern is concrete, checkable, and directly tied to the central claim.","tokens_in":18129,"tokens_out":12044,"duration_ms":116007,"concrete_test":"Check the public repository's preprocessing code to see whether the RQ-VAE training DataLoader is constructed from the full scBaseCount atlas or from the 80% per-cell-type training split defined in Appendix B. If it uses the full atlas, retrain the RQ-VAE on the 80% split only, regenerate token sequences for the same 22 grid points plus the four held-out runs, and refit Eq. 2 using the same Appendix D.3 procedure. If alpha, beta, or the four predicted losses shift by more than the reported residual scale (about 0.02 in log space), the leak is load-bearing and the held-out claim needs revision; if the refit reproduces Eq. 2 within that tolerance, the concern is benign and the conditional can be lifted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 and Appendix A.1 state the RQ-VAE tokenizer is trained once on the scBaseCount atlas and then frozen. Appendix B introduces the 20% per-cell-type validation split only for transformer runs ('no run ever trains on' those cells), and never states that tokenizer training was restricted to the 80% training pool. If the tokenizer saw the validation cells, L(N,D) in Eq. 2 is not a fully held-out loss: the learned encoder/decoder that produces the target tokens has been fit to the exact cells being scored. The same frozen tokenizer is used at every grid point, so the resulting bias need not cancel across N and D; a larger transformer could exploit tokenizer familiarity with validation cells more effectively, shifting alpha and the computed frontier D* ∝ C^0.59 vs N* ∝ C^0.41. The four held-out runs in Section 4.5 share the same tokenizer, so their 0.5% agreement cannot detect this leak. The magnitude of the bias is unknown, but the paper offers no argument or experiment ruling it out.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies conditional generation of single-cell gene expression vectors using a causal transformer over RQ-VAE discrete tokens. Its principal contribution is a joint two-exponent scaling law, L(N,D)=2.90+2.05e4/N^0.81+4.28e3/D^0.57, fit to a controlled 22-point N x D grid (1.31M-83.9M parameters, 38.1M-362M cells) and validated by four held-out runs predicted within 0.5%. The authors also evaluate generation quality against the pretraining loss, discuss why single-axis sweeps can look like data-scaling plateaus, and sketch a Stage 2 perturbation-response finetuning plan.","tokens_in":18310,"tokens_out":8601,"duration_ms":75134,"significance":"If the held-out status of the loss can be established, this is the first jointly fit two-exponent scaling law and compute-optimal frontier for a discrete-tokenized single-cell transformer. The paper's strengths are its controlled shape-preserving model sweep, independent N and D variation, proportional pool sampling, multi-start Approach-3 fitting with residual diagnostics, explicit four-point out-of-sample prediction, and released code. The framing against the diversity-over-scale literature is fair, and the limitations section is unusually candid. The two concerns below—tokenizer exposure to validation cells and ambiguity about the number of fit points—are central because both bear directly on whether L(N,D) in Eq. (2) is a true held-out loss and whether the reported R² and prediction errors support the stated law.","major_comments":[{"comment":"The validation split is introduced only for transformer training: Section 4.3 and Appendix B state that within each cell type 20% of cells are reserved and no transformer run trains on them, but Appendix A.1 says the RQ-VAE tokenizer is trained on the scBaseCount atlas and frozen, with no statement that validation cells were excluded from tokenizer training. Because L(N,D) in Eq. (2) is the cross-entropy on tokens generated by this tokenizer, any tokenizer exposure to validation cells makes the fitted loss partly in-sample, and the bias need not cancel across N and D since the same frozen tokenizer is used at every grid point. Please clarify whether validation cells were excluded from tokenizer training; if they were not, retrain the tokenizer on the 80% training pool and refit the law, and report the change in α and β.","section":"Section 4.3; Appendix A.1; Appendix B"},{"comment":"The paper is internally inconsistent about whether the four held-out runs are part of the 22 grid points used in the fit. Section 4.4 says the law is fit over the 22 (N,D) points and calls the R²=0.977 in-sample, while Section 4.5 says four runs were held completely out of the fit, and Appendix E.1 describes residuals over the 22 grid points. If the four runs are excluded from parameter estimation, the fit uses 18 points and the in-sample R² and residual diagnostics should be reported for those 18 points, with the four held-out runs reported only as prediction errors; if they are included, the label 'held out' is inaccurate. This distinction is load-bearing for the out-of-sample claim in Section 4.5.","section":"Section 4.4; Section 4.5; Appendix E.1"},{"comment":"The exponents α≈0.81 and β≈0.57, and consequently the frontier exponents D*∝C^0.59 and N*∝C^0.41 in Eq. (9), are reported without confidence intervals or standard errors. With five free parameters and at most 22 (or 18) points, the estimate depends on the initialization grid and on individual grid points; please provide bootstrap or refit-based intervals so the reader can judge whether the data-tilt of the frontier is statistically supported.","section":"Section 4.4; Eq. (2); Eq. (9)"}],"minor_comments":[{"comment":"The rendered full text contains numerous missing spaces and mangled words (e.g., 'Westudya' in the abstract and 'LLaMAarchitecture' in Section 3.4); please run a careful spelling and formatting pass before resubmission.","section":"Throughout"},{"comment":"The baseline values are reported as 0.964/0.038/0.998 in Table 1 but 0.9637/0.0382/0.9978 in Table 4; please reconcile the rounding consistently in the text and table captions.","section":"Section 4.1; Table 4"},{"comment":"Since C≈6ND counts D in cells rather than tokens, the constant offset is acknowledged in the Limitations, but stating explicitly in Section D.4 that this constant factor does not change the frontier exponents would avoid reader confusion.","section":"Section D.4"}],"recommendation":"major_revision","confidential_remarks":"The central scaling result is potentially valuable, but the tokenizer split issue must be settled before the held-out claim can be accepted. The inconsistency about the number of fit points is a simpler editorial problem but also needs a definitive correction. If the authors can show that validation cells were already excluded from tokenizer training, a careful revision with uncertainty quantification may suffice; otherwise, the additional tokenizer-retraining experiment should be feasible within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is a Chinchilla-style two-exponent scaling law, L(N,D) = 2.90 + 2.05e4/N^0.81 + 4.28e3/D^0.57, fit to a 22-point grid where model size and data budget are varied independently for a discrete-tokenized single-cell transformer. That is new for this subfield: prior work mostly sweeps one axis or confounds model size with other factors. The paper is transparent about reusing Hoffmann et al.'s Approach 3 fitting procedure, and the diagnostics are genuinely good: R^2 = 0.977, small symmetric residuals, and four completely held-out runs predicted within 0.5%. The generation-quality probe is also useful, showing that lower pretraining loss tracks better biological fidelity. I believe the central claim is solid within its stated scope, and the authors are careful to bound that scope in the limitations section.\n\nThe main soft spot is the tokenizer. The 20% validation split is introduced only for the transformer, and the RQ-VAE is described as \"trained on the scBaseCount atlas\" with no statement that validation cells were excluded from its training. If the tokenizer saw the validation cells, then the validation loss is partly in-sample, and the fitted exponents—especially the data exponent—could be biased. The four held-out runs share the same frozen tokenizer, so their agreement does not test this. This is not fatal, but it is load-bearing: the claim that this is a truly held-out loss needs either a clear statement that the tokenizer was trained on the 80% pool only, a re-fit with a split-aware tokenizer, or an auxiliary experiment showing the tokenizer's reconstruction loss does not differ between training and validation cells. The authors should also add uncertainty estimates for the exponents; a simple bootstrap would do. My only other complaint is a single sentence in the conclusion that locates other groups' plateaus as \"a downstream, not an upstream, phenomenon,\" which overreaches; the body's scope statement already says the right thing.\n\nWho this is for: practitioners building single-cell foundation models, especially those deciding how to allocate compute between parameters and data. The paper deserves a serious referee; the issues are concrete and fixable, and the result is useful. I would send it out.","headline":"A genuinely independent N×D scaling-law fit for a single-cell transformer, with strong held-out predictions; the unresolved tokenizer–validation-split question is the one load-bearing soft spot.","tokens_in":18872,"tokens_out":2032,"would_cite":true,"duration_ms":20409,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A causal transformer over discrete single-cell codes follows a joint two-exponent scaling law, with compute-optimal training tilted toward data.","keywords":["single-cell foundation model","scaling laws","autoregressive transformer","RQ-VAE tokenization","compute-optimal frontier","gene expression generation","Chinchilla Approach 3","self-supervised pretraining"],"falsifier":"Retrain the RQ-VAE on only the training split, excluding the 20 percent validation cells per cell type, and refit Eq. 2 on the same 22-point grid; the law is falsified if the exponents move beyond the reported fit uncertainty or if the four held-out predictions fall outside 0.5 percent.","tokens_in":17889,"feed_emoji":"🧬","tokens_out":10082,"duration_ms":78144,"temperature":0.7,"pith_summary":"The paper seeks to establish that a single-cell gene-expression generator—a causal transformer fed eight discrete tokens per cell from a learned residual-quantized VAE—obeys the same kind of joint scaling law as large language models. On a grid of 22 points where model size $N$ (1.31M–83.9M parameters) and data budget $D$ (38.1M–362M cells) vary independently, the pretraining cross-entropy fits $L(N,D)=2.90+2.05\\times10^4/N^{0.81}+4.28\\times10^3/D^{0.57}$ with both exponents positive and $R^2\\approx0.98$. Four runs held out of the fit are predicted within 0.5 percent, and the inverted law says compute-optimal training spends more on data than on parameters ($D^*\\propto C^{0.59}$, $N^*\\propto C^{0.41}$). If true, this turns single-cell foundation-model pretraining into a budget-allocation problem and reframes earlier no-data-scaling plateaus as capacity bottlenecks rather than inherent limits.","feed_headline":"Scaling law found for single-cell transformers: data wins","feed_subtitle":"A two-exponent fit predicts untouched runs within 0.5 percent and says extra cells beat extra parameters.","key_machinery":"The load-bearing mechanism is a tokenized sequence-modeling pipeline. An RQ-VAE compresses each 18,080-gene log-normalized expression vector into a 256-dimensional latent and quantizes it across eight residual codebooks of 256 entries each, turning one cell into a block of eight integer tokens; a causal transformer of the LLaMA family, trained from scratch by next-token cross-entropy, models sentences of 32 same-cell-type cells. The transformer adds a learned within-cell position embedding shared across cells and a learned bias separating cell tokens from separators, so the discrete structure of a cell is explicit. The scaling result comes from fitting the two-term parametric form by nonlinear least squares on log-residuals with a Huber loss and a multi-start initialization grid, then inverting the fit to obtain the compute-optimal model size $N^*(C)$.","core_discovery":"On the paper's own terms, the central discovery is that the self-supervised pretraining loss of an RQ-VAE-tokenized autoregressive transformer trained on single-cell expression follows the Chinchilla-style form $L(N,D)=E+A/N^\\alpha+B/D^\\beta$ over an independently varied $N\\times D$ grid, with fitted values $E\\approx2.90$, $A\\approx2.05\\times10^4$, $\\alpha\\approx0.81$, $B\\approx4.28\\times10^3$, and $\\beta\\approx0.57$. Both exponents are clearly positive; the in-sample fit has $R^2\\approx0.98$; and four runs deliberately excluded from the fit, one at each model size, are predicted within 0.5 percent. Because returns to parameters fall faster than returns to data, the compute-optimal frontier tilts toward data in this regime. The same model, conditioned on held-out vectors of a cell type, generates expression populations whose per-gene mean vectors sit just below empirical-replicate baselines (Pearson 0.960 vs 0.964, MAE 0.041 vs 0.038), with cell-type discriminability 0.990, and this generation quality tracks the pretraining loss across checkpoints.","pith_inferences":["The fitted floor $E\\approx2.90$ nats is partly set by the lossy RQ-VAE; one testable extension is to separate tokenizer reconstruction error from irreducible biological variability by re-fitting the law with more codebooks or larger codebooks.","The checkmark-shaped held-out loss on perturbation-benchmark control cells, which rises while the main validation loss falls, suggests that a perturbation-response finetune may need early checkpoint selection or corpus reweighting; this is the authors' open question, not a demonstrated result.","If the two-exponent form transfers to continuous-embedding single-cell models, scaling-law planning would generalize across architecture families; the paper does not test that transfer.","The law is fit in the 1.31M–83.9M parameter range, so extrapolation to billion-parameter scales is untested; a direct extension would add one large run and check whether $\\alpha$ and $\\beta$ hold."],"forward_implications":["The pretraining loss of a new run, and by proxy its generation quality, can be predicted before training from $N$ and $D$, so compute can be allocated to hit a target loss.","At the scales tested, adding cells is more compute-efficient than adding parameters: the optimal allocation follows $D^*\\propto C^{0.59}$ against $N^*\\propto C^{0.41}$.","A fixed-$N$ slice of the fitted surface plateaus as $D$ grows, which explains reported no-data-scaling results as model-capacity bottlenecks rather than a failure of scaling.","Generation fidelity tracks the pretraining loss monotonically (checkpoint correlations of $r=-0.99$ for Pearson and $r=+0.98$ for MAE), so the scaling law is a practical proxy for biological fidelity.","The law can be reproduced with modest compute, since the largest grid run is 83.9M parameters on a corpus of roughly 200M cells, making scaling-law planning accessible to small teams."],"supporting_citations":[{"why":"Provides the joint two-term power-law form $L(N,D)=E+A/N^\\alpha+B/D^\\beta$ and the Approach 3 fitting procedure reused here.","marker":"Hoffmann et al. (2022)"},{"why":"Establishes the original power-law scaling of transformer loss in model and data size that the paper extends to single cells.","marker":"Kaplan et al. (2020)"},{"why":"Supplies the residual vector quantization mechanism that turns each expression vector into eight discrete codes.","marker":"Lee et al. (2022)"},{"why":"Defines the LLaMA backbone architecture the transformer is built on.","marker":"Touvron et al. (2023)"},{"why":"Motivates the design of a learned quantized tokenizer paired with a causal transformer for sequence modeling.","marker":"Bai et al. (2023)"},{"why":"Is the diversity-over-scale study reporting no clear data-scaling law, which the paper reinterprets as a capacity bottleneck.","marker":"DenAdel et al. (2026)"},{"why":"Provides the single-axis N-only scaling evidence and saturation report that the joint grid is designed to adjudicate.","marker":"Wang et al. (2026a)"},{"why":"Supplies the position-paper evidence for context diversity over scale that the paper contrasts with its joint-scaling result.","marker":"Dibaeinia et al. (2026)"},{"why":"Supplies the learning-rate warmup and 10x decay schedule used for grid runs.","marker":"Rae et al. (2021)"},{"why":"Supplies the mean-pooled hidden-state mechanism used to extract cell-line context embeddings.","marker":"Zhang et al. (2024)"}],"fun_headline_variants":["Data beats parameters in single-cell transformer scaling","Two-exponent scaling law for single-cell generation","Single-cell transformer scaling: data-wins frontier","Scaling law predicts single-cell generation within 0.5%","Compute-optimal frontier tips toward data for single-cell models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the validation loss being measured on cells that neither the transformer nor the frozen tokenizer ever saw; the paper does not state that the tokenizer's training set excluded the per-cell-type validation cells.","fun_headline_variants_meta":{"raw":{"variants":["Data beats parameters in single-cell transformer scaling","Two-exponent scaling law for single-cell generation","Single-cell transformer scaling: data-wins frontier","Scaling law predicts single-cell generation within 0.5%","Compute-optimal frontier tips toward data for single-cell models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1579,"prompt_tokens":979,"completion_tokens":600,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":524}},"tokens_in":595,"tokens_out":600,"duration_ms":5574,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:54:28.137925+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the RQ-VAE on only the training split, excluding the 20 percent validation cells per cell type, and refit Eq. 2 on the same 22-point grid; the law is falsified if the exponents move beyond the reported fit uncertainty or if the four held-out predictions fall outside 0.5 percent.","supporting_citations":[],"review_version":1}