REVIEW 4 major objections 5 minor 33 references
NeuroPB: Scaling Neural Decoding with Pretrained Behavioral Representations
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read NeuroPB shows that a motor encoder pretrained on robotic trajectories, frozen, and used as an alignment target for macaque spike activity improves continuous trajectory decoding and scales with behavioral data.
desk verdict The core idea is fresh, but the headline result currently conflates pretraining with freezing; the scale/source comparisons are cleaner and deserve a control to fully land. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pretrained motor encoder, an eight-layer transformer trained by masked trajectory reconstruction on two-dimensional point trajectories and then frozen. It produces a normalized behavioral representation by pooling contextualized tokens, and this representation is the fixed target space for the neural side. The neural encoder is a Perceiver-style architecture that turns spike events into a fixed set of latent tokens with rotary positional embeddings, pools them, and projects them into the same space; a CLIP-style symmetric contrastive loss aligns temporally paired neural and behavioral representations while pushing mismatched pairs apart. A lightweight MLP decoder then reconstructs the trajectory from the aligned neural representation, and at inference the motor encoder is discarded. This three-stage design, pretrain a behavioral space, align neural activity to it, decode from the aligned representation, is what carries the argument.
What would settle it
Pretrain the motor encoder on robotic trajectories whose temporal order has been shuffled, or on random-walk point trajectories with the same marginal speed and target statistics; if decoding $R^2$ does not fall below Robot-Large-PT, then the benefit is not due to transferable kinematic structure. A complementary check is to rerun Macaque-PT with any trajectory from a session that appears in the evaluation split removed; if the performance gain disappears, leakage is the explanation.
Extended reading notes
Core claim
The central discovery is that a frozen behavioral representation space, pretrained without any neural data, can act as an anchor that organizes neural population activity for motor decoding. The paper shows that pretraining a transformer motor encoder on masked trajectory reconstruction, using either macaque reaches or robotic manipulation point tracks, produces a space into which spike activity can be aligned with a CLIP-style contrastive loss, and that this alignment is what drives the decoding gains. Removing the contrastive alignment to the pretrained space lowers center-out $R^2$ from 94.78% to 82.96%, the largest drop in the ablations. Equally central is the scaling result: a motor encoder pretrained on LIBERO-100, containing 5,000 robotic trajectories, outperforms both a scale-matched robotic pretraining with 500 trajectories and macaque pretraining, even though the robotic data come from a different embodiment and task distribution. The authors take this as evidence for transferable kinematic structure shared across biological and artificial movement systems, and as a demonstration that behavioral data can substitute for scarce neural recordings in pretraining.
Load-bearing premise
The claim rests on the assumption that a motor encoder trained only on two-dimensional point trajectories from robots captures kinematic structure that is compatible with macaque motor cortex activity, so that freezing that space and aligning spikes to it helps rather than distorts decoding.
Editorial extensions
If this is right
- Behavioral pretraining is a scalable axis: with neural data held fixed, decoding accuracy improves as robotic pretraining data grow from 500 to 5,000 trajectories, so adding more behavior data should continue to help.
- Calibration requirements drop sharply: with only 10% of target-session data, the pretrained model matches the scratch model trained on 100% of the data across session, subject, and task transfer settings.
- The pretrained behavioral space suppresses session-specific nuisance variation: after alignment, neural representations cluster by reach direction rather than by recording session, supporting better cross-session generalization.
- The motor encoder is not needed at deployment, so the added pretraining cost is paid once offline; the deployed model is just the neural encoder plus a light decoder.
Reading between the lines
- If the scaling trend extends beyond 5,000 trajectories, behavioral pretraining on much larger embodied datasets could push neural decoding toward plug-and-play calibration for new sessions and subjects, a regime the paper does not test.
- Because the target space is purely kinematic, the same alignment recipe could be applied to non-invasive recordings such as EEG or fMRI, provided those signals carry enough movement-related information; the paper only evaluates invasive macaque data.
- A direct extension would be to ablate the content of the behavioral representation, for example pretraining on speed-only or angle-only trajectories, to identify which kinematic features the neural encoder actually latches onto; the current study compares sources and scales but not feature content.
- The paper does not disclose the session provenance of the macaque trajectories used for Macaque-PT; verifying that they do not overlap with evaluation sessions is an open check, and the robotic-pretraining results are independent of that provenance.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces NeuroPB, a three-stage framework for neural trajectory decoding. A transformer-based motor encoder is first pretrained on behavioral trajectories (macaque reaching trajectories, scale-matched robotic trajectories from LIBERO-Spatial, or large-scale robotic trajectories from LIBERO-100) using masked trajectory reconstruction. The pretrained motor encoder is frozen, and a Perceiver-based neural encoder is trained with a CLIP-style symmetric contrastive loss to align spike-activity representations with the frozen behavioral representation space, while a lightweight MLP decoder reconstructs positions from the aligned neural representation. The method is evaluated on macaque M1/PMd recordings from the Perich, Gallego, and Miller dataset, on center-out and random-target tasks. The authors report state-of-the-art single-session performance, an 11% R2 improvement over a randomly initialized and jointly optimized 'Scratch' motor encoder on center-out and 8% on random-target, comparable performance between Macaque-PT and Robot-Matched-PT, further gains with Robot-Large-PT, improved cross-session/cross-subject/cross-task generalization, and calibration efficiency requiring only about 10% of target data to match Scratch trained on 100% of the data.
Significance. If the reported effects are genuine, the paper identifies a potentially scalable alternative to neural-only pretraining: using large, readily available behavioral or robotic trajectory datasets to define a frozen representation space that shapes a neural encoder for motor decoding. This would be a useful contribution to the BCI literature. The paper has several strengths: it uses publicly available macaque data, provides ablations of key neural-encoder components (Table 2), includes a clear statement that the scale illustration in Fig. 4 is conceptual rather than measured, and compares two pretraining conditions (Macaque-PT and Robot-Matched-PT) at matched data scale. The central causal claim, however, is currently under-supported because the main pretrained-vs-scratch comparison conflates two variables, and the calibration-efficiency analysis uses asymmetric optimization protocols. The robot-vs-macaque source comparison is better controlled but still lacks a frozen-random or non-informative-pretraining baseline.
major comments (4)
- [Method, 'Motor Encoder'; Experiments, 'Effect of Pretrained Motor Encoder'] The central comparison between pretrained and Scratch motor encoders conflates pretraining with the freezing protocol. The paper states that in Scratch the motor encoder is randomly initialized and jointly optimized during neural alignment, whereas Macaque-PT, Robot-Matched-PT, and Robot-Large-PT are kept frozen during neural alignment. Thus the reported 11% R2 increase on CO and 8% on RT could reflect the stabilizing effect of a fixed, nontrivial representation target rather than transferable behavioral knowledge. To support the central claim, the authors should add a control condition with a randomly initialized but frozen motor encoder, and ideally a motor encoder pretrained on shuffled or non-informative trajectories and frozen. Without such controls, the 'behavioral pretraining is a critical component' conclusion in the Effect of Pretrained Motor Encoder section is not justified.
- [Experiments, 'Calibration-Efficiency Validation'; Appendix, 'Calibration Details'] The calibration-efficiency comparison is asymmetric. In the transfer experiments, pretrained models optimize only newly introduced unit and session embeddings on the calibration subset, while the Scratch baseline 'initialized the entire model randomly and optimized all components using each calibration subset.' This heavily disadvantages Scratch in the low-data regime and makes the claim that 'only 10% calibration needed to match training from scratch' an artifact of the comparison protocol rather than a demonstration of behavioral transfer. The authors should either train Scratch under the same frozen-encoder protocol (with a frozen random encoder) or optimize all components of the pretrained models during calibration, so the amount of learned plasticity is matched.
- [Experiments, 'Comparison with Baselines'; Table 1] Table 1 reports NEDS, NDT2, and POYO results obtained from other publications (Ryoo et al. 2026; Mao et al. 2026) rather than re-run under the same data-splitting and evaluation protocol. Because the authors explicitly state that the classical baselines were 'reproduced using the same protocol' but do not do so for these three neural-decoding baselines, the state-of-the-art claim on both CO and RT is not established under a controlled comparison. Different split conventions, session filtering, or window overlap could easily explain part of the reported gap, particularly the RT improvement from 0.7156 (POYO) to 0.8475 (NeuroPB). The authors should either re-run these baselines with their own protocol or explicitly restrict the SOTA claim to the reproduced baselines.
- [Method, 'Scaled Pretrained Motor Encoder'; Experiments, 'Datasets'] The source of the macaque trajectories used for Macaque-PT is never disclosed. If these trajectories come from the same dataset or, more seriously, from the same recording sessions used for neural-decoding evaluation, the Macaque-PT results could benefit from information leakage, and the robot-vs-macaque comparison would not be clean. The authors should state exactly how macaque trajectories were obtained, how many trajectories/sessions they comprise, and confirm that all pretraining data are disjoint from the evaluation sessions at the session level.
minor comments (5)
- [Various sections] The manuscript contains several grammatical errors that should be corrected in a language pass. Examples include 'For adapt to different numbers of spikes varies across windows' in the Neural Encoder section, 'The output of the final self-attention block as the neural representation' (missing verb), and 'multi-layer perception' in the Appendix.
- [Figure 4 and Appendix, 'Details of robotic pretraining datasets'] The left panel of Fig. 4 is labeled as a conceptual illustration of relative pretraining-data scales and is explicitly not derived from measured scores. This is an honest disclosure, but a quantitative table listing the number of trajectories, total time steps, and data diversity metrics for Macaque-PT, Robot-Matched-PT, and Robot-Large-PT would be more informative than a conceptual schematic.
- [Table 1] Table 1 reports R2 values with standard deviations but does not include the number of sessions or a significance test across methods. Given the small number of datasets/conditions, the authors should clarify how many sessions each mean covers and whether the differences between NeuroPB and the best baseline are statistically significant.
- [Appendix, 'Additional Results'] The heading 'Generation Result' appears to be a typo for 'Generalization Result.' Please correct it.
- [Visualization Analysis] The PCA visualization in Fig. 6B is qualitative. Since the paper claims that alignment 'suppresses session-dependent nuisance variation while preserving behaviorally relevant movement information,' a quantitative metric (e.g., silhouette score computed separately for direction labels and session labels) would strengthen this claim.
Circularity Check
No significant circularity: NeuroPB is an empirical evaluation whose central claims are supported by held-out decoding scores rather than by self-citation or definition.
full rationale
NeuroPB is an empirical evaluation, not a first-principles derivation, so the circularity burden is low. The motor encoder is pretrained on macaque or LIBERO trajectories via masked reconstruction and then frozen; the neural encoder and motor decoder are optimized on paired neural-behavioral data using a contrastive objective and an MSE trajectory loss. The reported improvements, including the 11% R2 gain on CO and 8% on RT, are measured test-set differences between initialization conditions, not fitted parameters renamed as predictions. No equation defines the decoding output in terms of the pretraining objective by construction, and no uniqueness theorem or author-loaded constraint is invoked to force the representation choice. The only author self-citation (Song et al. 2023) appears in related work on EEG image decoding and is not load-bearing for any central claim. The Scratch baseline differs from the pretrained conditions both in initialization and in whether the motor encoder is frozen; this is an experimental confound relevant to interpreting the causal effect of pretraining, but it is not a circularity because the improvement does not follow from the setup by definition. The paper's stated limitation—offline evaluation using invasive macaque recordings—is a scope limitation, not a circular step. The undisclosed source of Macaque-PT trajectories is a missing-detail concern but does not exhibit a specific reduction of a claimed result to its inputs. Therefore, no circular step is identified.
Assumptions & free parameters
free parameters (2)
- contrastive temperature tau =
learnable, initialized to 0.07
- loss weights for L_align and L_motor =
both 1.0
assumptions (4)
- domain assumption Neural population activity in M1/PMd contains information sufficient to reconstruct continuous arm trajectories, and a frozen behavioral representation space can serve as a useful alignment target.
- domain assumption Biological and artificial movement systems share transferable kinematic structure, so robotic trajectories can provide a behavioral prior for macaque reaching.
- domain assumption Averaging (pooling) contextualized trajectory tokens produces a behaviorally meaningful representation of an entire trajectory segment.
- standard math Standard transformer attention and rotary position embeddings adequately model spike-event sequences.
Cite this review
Pith. "Pith review of NeuroPB: Scaling Neural Decoding with Pretrained Behavioral Representations." pith.science (2026). https://pith.science/paper/COSHVUI7
@misc{pith2026260804389,
author = {Pith},
title = {Pith review of: NeuroPB: Scaling Neural Decoding with Pretrained Behavioral Representations},
year = {2026},
howpublished = {\url{https://pith.science/paper/COSHVUI7}},
note = {Machine review of arXiv:2608.04389}
}
abstract
Decoding continuous motor trajectories from neural activity is essential for developing practical brain-computer interfaces (BCIs). However, current neural decoders are constrained by the limited scale and heterogeneity of neural recordings. In contrast, behavioral data can be collected more readily and at substantially larger scale from humans, animals, simulations, and robotic systems. Here, we introduce NeuroPB, a framework that scales neural decoding by transferring knowledge from pretrained behavioral representations. NeuroPB first pretrains a motor encoder on large-scale motor behavior data and then aligns neural activity with the resulting behavioral representation space using a limited set of paired neural-behavioral recordings. A neural encoder and lightweight motor decoder are subsequently optimized to reconstruct continuous movement from the aligned neural representations. Across multiple macaque motor datasets, behavioral pretraining improves trajectory decoding, including an 11% $R^2$ increase on center-out and 8% on random-target compared with training the motor encoder from scratch. Notably, pretraining on robotic trajectories achieves performance comparable to pretraining on macaque trajectories, demonstrating that transferable kinematic structure is shared across biological and artificial models. Moreover, decoding performance improves as the scale and diversity of robotic pretraining data increase, when the amount of neural data is fixed. Pretraining also enhances generalization across recording sessions, subjects, and motor tasks, with only 10% calibration needed to match training from scratch. Overall, these results establish behavioral pretraining as a scalable source for neural decoding and provide a promising route toward high-performance and calibration-efficient BCIs under limited neural data.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
A high-performance brain--computer interface for finger decoding and quadcopter game control in an individual with paralysis , author=. Nature Medicine , pages=. 2025 , publisher=
work page 2025
-
[2]
Nature machine intelligence , volume=
Brain--computer interface control with artificial intelligence copilots , author=. Nature machine intelligence , volume=. 2025 , publisher=
work page 2025
-
[3]
A high-performance brain--computer interface for finger decoding and quadcopter game control in an individual with paralysis , author=. Nature Medicine , volume=. 2025 , publisher=
work page 2025
-
[4]
Learning to control a brain--machine interface for reaching and grasping by primates , author=. PLoS Biology , volume=. 2003 , publisher=
work page 2003
-
[5]
Inferring single-trial neural population dynamics using sequential auto-encoders , author=. Nature Methods , volume=. 2018 , publisher=
work page 2018
-
[6]
Advances in Neural Information Processing Systems , volume=
A unified, scalable framework for neural population decoding , author=. Advances in Neural Information Processing Systems , volume=
-
[7]
Machine learning for neural decoding , author=. eneuro , volume=. 2020 , publisher=
work page 2020
-
[8]
Advances in Neural Information Processing Systems , volume=
Neural data transformer 2: multi-context pretraining for neural spiking activity , author=. Advances in Neural Information Processing Systems , volume=
Show all 33 references
-
[9]
ArXiv , pages=
Neural encoding and decoding at scale , author=. ArXiv , pages=
-
[10]
arXiv preprint arXiv:2401.00025 , year=
Any-point trajectory modeling for policy learning , author=. arXiv preprint arXiv:2401.00025 , year=
-
[11]
Neurocomputing , volume=
Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=
2024
-
[12]
arXiv preprint arXiv:2107.14795 , year=
Perceiver io: A general architecture for structured inputs & outputs , author=. arXiv preprint arXiv:2107.14795 , year=
-
[13]
International Conference on Machine Learning , pages=
Learning transferable visual models from natural language supervision , author=. International Conference on Machine Learning , pages=. 2021 , organization=
2021
-
[14]
Nature methods , volume=
A large-scale neural network training framework for generalized estimation of single-trial population dynamics , author=. Nature methods , volume=. 2022 , publisher=
2022
-
[15]
Nature , volume=
Learnable latent embeddings for joint behavioural and neural analysis , author=. Nature , volume=. 2023 , publisher=
2023
-
[16]
Nature Biomedical Engineering , volume=
A cryptography-based approach for movement decoding , author=. Nature Biomedical Engineering , volume=. 2017 , publisher=
2017
-
[17]
International Conference on Learning Representations , volume=
Multi-session, multi-task neural decoding from distinct cell-types and brain regions , author=. International Conference on Learning Representations , volume=
-
[18]
Advances in Neural Information Processing Systems , volume=
Libero: Benchmarking knowledge transfer for lifelong robot learning , author=. Advances in Neural Information Processing Systems , volume=
-
[19]
2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=
Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0 , author=. 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2024 , organization=
2024
-
[20]
Conference on Robot Learning , pages=
Bridgedata v2: A dataset for robot learning at scale , author=. Conference on Robot Learning , pages=. 2023 , organization=
2023
-
[21]
Experimental brain research , volume=
Deciding when and how to correct a movement: discrete submovements as a decision making process , author=. Experimental brain research , volume=. 2007 , publisher=
2007
-
[22]
Nature communications , volume=
Hierarchical motor control in mammals and machines , author=. Nature communications , volume=. 2019 , publisher=
2019
-
[23]
arXiv preprint arXiv:2108.01210 , year=
Representation learning for neural population activity with neural data transformers , author=. arXiv preprint arXiv:2108.01210 , year=
-
[24]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Multi-dimensional neural decoding with orthogonal representations for brain-computer interfaces , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[25]
Advances in Neural Information Processing Systems , volume=
A generalist intracortical motor decoder , author=. Advances in Neural Information Processing Systems , volume=
-
[26]
Nature Machine Intelligence , volume=
Decoding speech perception from non-invasive brain recordings , author=. Nature Machine Intelligence , volume=. 2023 , publisher=
2023
-
[27]
The Fourteenth International Conference on Learning Representations , year=
A cross-species neural foundation model for end-to-end speech decoding , author=. The Fourteenth International Conference on Learning Representations , year=
-
[28]
Decoding natural images from
Song, Yonghao and Liu, Bingchuan and Li, Xiang and Shi, Nanlin and Wang, Yijun and Gao, Xiaorong , journal=. Decoding natural images from
-
[29]
2024 , organization=
Xia, Weihao and de Charette, Raoul and Oztireli, Cengiz and Xue, Jing-Hao , booktitle=. 2024 , organization=
2024
-
[30]
Proceedings of the AAAI Conference on Artificial Intelligence , volume=
Neurobridge: Bio-inspired self-supervised eeg-to-image decoding via cognitive priors and bidirectional semantic alignment , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
-
[31]
Neuron , volume=
A neural population mechanism for rapid learning , author=. Neuron , volume=. 2018 , publisher=
2018
-
[32]
Advances in Neural Information Processing Systems , volume=
Generalizable, real-time neural decoding with hybrid state-space models , author=. Advances in Neural Information Processing Systems , volume=
-
[33]
arXiv preprint arXiv:2607.14086 , year=
Leveraging unlabelled data for generalizable neural population decoding , author=. arXiv preprint arXiv:2607.14086 , year=
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.