REVIEW 3 major objections 5 minor 13 references
A continuous autoregressive transformer pretrained on mouse calcium imaging, with its backbone frozen, generalizes to new mouse, larval zebrafish, and C. elegans datasets for neural forecasting and behavior decoding.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-07-31 23:54 UTC pith:AYSR5TMV
load-bearing objection Useful cross-species transfer benchmark, but the 'consistently outperforms' claim needs error bars and significance tests before I'd trust the headline. the 3 major comments →
CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
CAPT's central claim is that a single continuous autoregressive transformer, pretrained with mean-squared-error next-patch prediction on one mouse calcium-imaging dataset, produces representations reusable across datasets, paradigms, and species. The authors show that freezing the pretrained backbone and updating only trainable neuron/session embeddings (for forecasting) or a task head (for behavior decoding) consistently outperforms specialized forecasting and decoding models, including on larval zebrafish and C. elegans, whose recording rates, neuron counts, and anatomy differ substantially from the mouse source. Additionally, linear classifiers on CAPT neuron embeddings recover NeuroPAL-d
What carries the argument
Continuous patch tokenization: each neuron trace is cut into non-overlapping patches of length L=8 samples and projected directly into model space with a linear layer plus learnable neuron and session embeddings; the Dual-axis Transformer processes them with causal temporal attention across time and neuron-axis attention across the population, and predicts the next continuous patch in raw trace space under an MSE loss. Two corruptions (pseudo scheduled sampling and temporal neighborhood replacement) make training robust to imperfect histories. The same continuous tokens are used for pretraining, forecasting, and as input to a nonlinear low-rank per-neuron readout for behavior decoding; keepi
Load-bearing premise
The fixed patch length of 8 samples is assumed to give temporally commensurate tokens across datasets, even though it spans roughly 0.5 seconds of mouse data, 0.8–5 seconds of zebrafish data, and about 4.8 seconds of C. elegans data; if normalized calcium dynamics are not scale-invariant across those durations, the frozen temporal attention weights lose their meaning.
What would settle it
Retrain CAPT with per-dataset patch lengths chosen to equalize real time per patch, and compare with the fixed L=8 result on larval zebrafish and C. elegans forecasting. If per-dataset patch alignment substantially improves or fixed L=8 fails on a new species at a different sampling rate, the cross-species transfer claim rests on a coincidental token size rather than a general mechanism.
If this is right
- If true, a single frozen backbone can serve as a general feature extractor for calcium imaging, so new datasets require only small per-session or per-neuron embeddings rather than full retraining.
- Cross-species transfer would mean shared low-level dynamics across mouse, fish, and worm calcium signals are captured by the same temporal attention weights, opening the door to multi-species pretraining.
- Because embeddings line up with anatomical cell identity across datasets, the same functional coordinates could link calcium activity to connectomic and transcriptomic atlases in C. elegans and beyond.
- The simple MSE objective removes the need for a separate vector-quantization tokenizer, reducing hyperparameters and keeping forecasting in the trace domain.
- The authors note transfer gains are smaller than in-domain gains, implying the source distribution still constrains the backbone and that wider pretraining data is the next step.
Where Pith is reading between the lines
- A testable extension: fix patch length per dataset so each patch covers the same real time rather than a constant L=8; if performance survives, the mechanism is scale-invariant calcium dynamics, and if not, the fixed-token assumption is load-bearing.
- If the shared embedding space holds, one could predict NeuroPAL identities for unannotated neurons by nearest-neighbor transfer, effectively using CAPT as a functional cell-identity mapper.
- Because the decoder readout is low-rank per neuron, behavior decoding performance likely depends on global neuron ordering and alignment; comparing with permutation-invariant readouts would isolate whether population-level structure or per-neuron alignment drives transfer.
- CAPT embeddings could provide a common latent space for cross-species registration of neural populations, a step the paper gestures at but does not implement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CAPT, a continuous autoregressive transformer for calcium imaging population dynamics. CAPT tokenizes single-neuron calcium traces into continuous temporal patches, projects them with neuron/session embeddings, and trains a dual-axis transformer with an MSE next-patch objective. After pretraining on a large mouse dataset (Tseng et al.), the backbone is frozen and only dataset-specific embeddings or decoding heads are trained for transfer. The authors evaluate CAPT on neural population forecasting and behavior decoding across eight external mouse, zebrafish, and C. elegans datasets, comparing with POCO, POYO+, and CalM. They report that CAPT consistently outperforms the baselines, and additionally show that CAPT embeddings support NeuroPAL-based neuron identity classification across C. elegans datasets. The paper includes detailed dataset descriptions, hyperparameter tables, ablations, and additional results in appendices.
Significance. If the reported transfer results are statistically robust, the paper would be a valuable step toward general-purpose calcium-imaging foundation models: the frozen-backbone transfer protocol is clean, the benchmark spans three species and multiple laboratories, and the NeuroPAL embedding analysis is a useful extension beyond predictive accuracy. The authors provide extensive implementation detail, consistent preprocessing, and ablate key design choices. However, the central comparative claim currently rests on point estimates without uncertainty quantification, and the fixed patch-length tokenization raises a cross-species architectural concern. These issues are fixable within the scope of the paper and do not invalidate the approach, but they must be addressed before the headline claim can be accepted.
major comments (3)
- [§5.1–5.4, Figs. 2–5] The central claim that CAPT 'consistently outperforms' baselines is supported only by point estimates. The main transfer comparisons in Figs. 3 and 5, and the embedding classification in Fig. 4, lack error bars, number of seeds, and significance tests. Several margins are very small; for example, Brysch forecasting correlation is 0.1895 for CAPT versus 0.1872 for CalM. Appendix A describes a hierarchical aggregation (per neuron, per trial, per session) that could support paired tests, and Appendix E reports standard deviations for only one comparison (Table 11), but not for the main tables. The limitation paragraph in §6 also acknowledges that transfer gains are generally smaller, making the missing uncertainty quantification more consequential. Please add confidence intervals or error bars, report the number of seeds, and provide paired significance tests with multiple-comparison correc
- [§3.1, Appendix D, Table 1] The patch length L=8 is used for all datasets, but it is selected on 6 Hz mouse pretraining data (Appendix D). At target sampling rates, the same L corresponds to very different real-time windows: about 0.5 s at 10 Hz (Sun) and about 4.8 s at 1.67 Hz (C. elegans). The frozen temporal attention is therefore asked to operate over substantially different temporal receptive fields. The ablation in Table 9 only varies L on the 6 Hz source subset and does not test whether L=8 is appropriate for 1.67–15.6 Hz target data. This is load-bearing for the cross-species transfer claim because the continuous patch tokenization is the main mechanism that makes the traces commensurable. Please either adapt L to each target dataset's sampling rate, add a per-dataset patch-length sensitivity analysis, or explicitly show that the real-time span of a patch does not affect the transfer conclusions.
- [§4.3, Appendix B, Appendix E, Fig. 5] Baseline comparability is not fully established. POCO is restricted to fixed context/horizon of 40/24 steps (Appendix B), and for the Lavian whole-brain dataset it is reported as 'unavailable' because the downstream setup is incompatible (Appendix E). Thus the claim that CAPT outperforms specialized baselines on all transfer datasets is partly a consequence of protocol constraints rather than model quality alone. Similarly, POYO+ fine-tuning only adjusts sequence length and latent step based on real time; Table 11 shows that in-domain POYO+ performs much better than cross-species fine-tuning, which supports the authors' distribution-shift interpretation but does not fully establish that the fine-tuning protocol is equally favorable to all methods. Please either evaluate all baselines under the same transfer protocol where possible, or clearly qualify the comparisons for which a baseline
minor comments (5)
- [Appendix B, POCO paragraph] Typo: the text says 'we freeze the POYO backbone' but should say 'POCO backbone'.
- [Fig. 3E/F] The decoding R2 axis is shown from 0 to 1, but the text reports negative values such as -0.0124 for POYO+ on Atanas head-angle decoding. The negative bars are likely clipped or invisible; please adjust the axis or annotate the values.
- [§5.4, Fig. 4B] The embedding classification results would be easier to interpret with details on SVM hyperparameters, class sizes, and whether the 5-fold CV is stratified; also, the 'chance' levels should state whether they are based on label permutation or majority-class proportion.
- [Table 9] The ablation table reports mean±std but not the number of sessions or seeds; adding the number of runs would help assess the stability of the patch-length comparison.
- [§2.2] The related work section introduces 'POCO' for forecasting and 'POYO+' for decoding, but the names are close; a brief clarifying sentence distinguishing them would improve readability.
Circularity Check
No circular derivation; central transfer claims rest on external benchmarks. Minor self-citation to CalM/DAT backbone is not load-bearing.
full rationale
The paper's derivation chain is empirical rather than definitional. CAPT is pretrained with the autoregressive MSE objective in Eq. (6), which predicts future continuous patches from past patches, and transfer is tested on eight independent datasets with the pretrained backbone frozen and only neuron/session embeddings or decoding heads updated. The central claim—that CAPT consistently outperforms specialized and general-purpose baselines on cross-dataset and cross-species forecasting and decoding—is evaluated on external held-out data, not derived from the training objective. No equation reduces a reported prediction to a fitted input: patch length L=8 is chosen by ablation, and the NeuroPAL identity analysis uses labels that were not part of forecasting or decoding training. The only notable self-citation is the DAT backbone and evaluation protocol taken from CalM [Xu et al., 2026], which shares two authors with this paper; however, CAPT's contribution is a different continuous patch tokenization and a continuous autoregressive objective, and the transfer evidence is independent of that citation. Thus the self-citation is not load-bearing. Statistical concerns raised by the skeptical reader—such as the absence of error bars or significance tests on small margins like Brysch 0.1895 vs 0.1872—are evidentiary limitations, not circularity.
Axiom & Free-Parameter Ledger
free parameters (5)
- patch_length L =
8
- scheduled_sampling_max_probability =
0.6
- neighborhood_replacement_probability =
0.1
- decoder_rank =
16
- temporal_smoothing_kernel_size =
9
axioms (5)
- domain assumption Z-scored, causally smoothed calcium traces from different species preserve shared temporal structure sufficient for cross-species transfer
- domain assumption Pseudo-trial segmentation with temporal splits and block-random splits yields independent train/test sets without leakage
- domain assumption Freezing the backbone and adapting only embeddings/heads is a meaningful test of representation transfer
- domain assumption The dual-axis transformer architecture from CalM is an appropriate inductive bias for population dynamics
- domain assumption NeuroPAL labels are a valid ground truth for cell identity, and linear SVM on frozen embeddings reveals representational structure
read the original abstract
Large-scale calcium imaging has created an opportunity to build foundation-style models for neural population dynamics, but a central question remains unresolved: \textbf{whether a model pretrained on one collection of recordings can generalize to new datasets, experimental paradigms, and even species.} Existing approaches are often designed for specific tasks and evaluated on a single dataset, making it unclear whether their learned representations are reusable for new calcium trace datasets. To tackle this gap, we present \textbf{CAPT}, a \textbf{C}ontinuous \textbf{A}utoregressive \textbf{P}opulation \textbf{T}ransformer for calcium population dynamics. CAPT models continuous calcium traces directly through a continuous patch tokenization strategy and is trained autoregressively, enabling end-to-end pretraining and adaptation to diverse downstream tasks. We first pretrain CAPT on a large-scale mouse calcium imaging dataset and evaluate its transferability across independent mouse, larval zebrafish, and \textit{C. elegans} datasets collected by different laboratories. In these transfer settings, the pretrained backbone is frozen and only adaptation modules are updated. Across neural population forecasting and behavior decoding tasks, CAPT consistently outperforms specialized and general-purpose baselines. Alongside predictive performance, multimodal analyses using NeuroPAL annotations in \textit{C. elegans} datasets show that CAPT embeddings form a shared functional space across datasets and capture anatomical cell-identity-related structure. These results suggest that the continuous autoregressive modeling opens up possibilities for a simple route towards general-purpose neural foundation models for calcium imaging, which can generalize across datasets, experimental paradigms, and species.
Figures
Reference graph
Works this paper leans on
-
[2]
During fine-tuning, we freeze the POYO backbone and the conditioning MLP, and only update the neural and session embeddings. Table 2: Hyperparameters used for training POCO model Hyperparameter Value MLP hidden size 1024 Decoder layers 1 Decoder hidden size 128 Number of heads 16 Latent tokens 8 FFN hidden size 1024 Compression Factor 8 POYO+ For POYO+ [A...
2025
-
[3]
The pretrain backbone is kept frozen across all fine-tuning experiments
For cross-dataset fine-tuning, we only adjust the sequence length and latent step based on real time to match sampling rates across different datasets. The pretrain backbone is kept frozen across all fine-tuning experiments. For within-dataset fine-tuning, the decoder head is frozen, and only the neural and session embeddings are updated. For cross-datase...
2026
-
[5]
Konstantin F Willeke, Polina Turishcheva, Alex Gilbert, Goirik Chakrabarty, Hasan A Bedel, Paul G Fahey, Yongrong Qiu, Marissa A Weis, Michaela Vystrˇcilová, Taliah Muhammad, et al. Omni- mouse: Scaling properties of multi-modal, multi-task brain models on 150b neural tokens.arXiv preprint arXiv:2604.18827,
-
[6]
Continuous calcium traces are divided into non-overlapping temporal patches, and the model predicts the next continuous patch directly in the original trace space
C Model and hyperparameters For CAPT, we pretrain a continuous patch-based autoregressive backbone on the multi-session forecasting task. Continuous calcium traces are divided into non-overlapping temporal patches, and the model predicts the next continuous patch directly in the original trace space. We use mean squared error as the forecasting objective....
2048
-
[7]
Xinhong Xu, Yimeng Zhang, Qichen Qian, and Yuanlong Zhang. Self-supervised foundation model for calcium-imaging population dynamics.arXiv preprint arXiv:2604.04958,
-
[8]
Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers.arXiv preprint arXiv:2108.01210,
-
[13]
We train for 200 epochs during pretraining and all fine-tuning tasks and the checkpoint with the best validation Pearson correlation is selected
For forecasting and decoding fine-tuning, we adopt the same fine-tuning strategy as described above for CalM. We train for 200 epochs during pretraining and all fine-tuning tasks and the checkpoint with the best validation Pearson correlation is selected. Table 7: Hyperparameters used for CAPT backbone Hyperparameter Value Patch length 8 Model dimension 5...
2048
-
[2006]
Ji Xia, Yizi Zhang, Shuqi Wang, Genevera I Allen, Liam Paninski, Cole Lincoln Hurwitz, and Kenneth D Miller. Inpainting the neural picture: Inferring unrecorded brain area dynamics from multi-animal datasets.arXiv preprint arXiv:2510.11924,
-
[2011]
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426,
-
[2013]
Antonis Antoniades, Yiyi Yu, Joseph Canzano, William Wang, and Spencer LaVere Smith. Neu- roformer: Multimodal and multitask generative pretraining for brain data.arXiv preprint arXiv:2311.00136,
-
[2022]
elegans) and covering diverse experimental paradigms
12 A Dataset In this work, we use a collection of calcium imaging datasets spanning three species (mouse, larval zebrafish, andC. elegans) and covering diverse experimental paradigms. In total, the datasets comprise 676,424 neurons and 530 sessions. All neural traces are smoothed and z-score normalized, while behavioral variables are z-score normalized be...
2022
-
[2025]
Sangyoon Bae, Mehdi Azabou, Jiook Cha, and Blake Richards. Decoding dynamic visual experience from calcium imaging via cell-pattern-aware ssl.arXiv preprint arXiv:2510.18516,
-
[2026]
Christian Brysch, Claire Leyden, and Aristides B
URL https://datadryad.org/dataset/doi: 10.5061/dryad.w9ghx3g4v. Christian Brysch, Claire Leyden, and Aristides B. Arrenberg. Functional architecture underlying binocular coordination of eye position and velocity in the larval zebrafish hindbrain.BMC Biology, 17(1):110,
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.