{"id":"d0aae09a-2bfc-40a5-aea0-6d6e55e3d219","arxiv_id":"2507.02264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Off-the-shelf LLMs, especially DeepSeek Coder-7b, decode tail behavior from whole-brain calcium imaging in zebrafish better than RNN baselines, but with weak absolute accuracy.","lead":"What if large language models, built for text, could read brain-wide neural activity and predict an animal's movements? This paper tests that idea in larval zebrafish and finds that a mixture-of-experts model, DeepSeek Coder-7b, tracks tail movements better than recurrent networks, though the overall match is still weak.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Overlapping sliding windows are treated as independent in the Mann-Whitney tests that establish DeepSeek-c7b's superiority; with effective sample size inflated, the headline ranking may not be significant.","rationale":"The reader's stated weakest assumption concerns uniform hyperparameters across model classes; that is a real fairness concern, but the more load-bearing issue is statistical validity: the significance tests that establish DeepSeek-c7b's superiority treat overlapping windows as independent. The reader's rationale does mention this issue, so there is partial agreement, but it is not the reader's identified weakest assumption. The paper's strongest empirical result is a small RMSE advantage with overlapping-window SEMs and Mann-Whitney tests; if the overlap is not accounted for, the p-values are likely inflated and the central ranking may rest on an artifact. The proposed concrete test—reanalyzing with non-overlapping windows or block bootstrap—would directly settle whether the reported advantages survive a valid dependence structure. The paper does provide code and data, which is a genuine strength, and the salience analyses are interesting, but they do not rescue the significance claim. Therefore the appropriate verdict remains CONDITIONAL, with the condition being a valid statistical reanalysis of the model comparison.","tokens_in":17075,"tokens_out":4448,"duration_ms":54082,"concrete_test":"Recompute the LSTM-versus-DeepSeek comparison in Table 1 using only test windows separated by at least s frames (non-overlapping windows), or a block bootstrap with block length s clustered by fish. If the p-value for any sequence length rises above 0.05, the claimed significant improvement is not supported and the headline ranking should be treated as provisional.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that DeepSeek-c7b significantly outperforms the LSTM (Table 1, asterisks) rests on Mann-Whitney U tests applied to per-sequence RMSE values from overlapping sliding windows. Each test sequence is a window of length s, and adjacent windows overlap by s-1 frames (e.g., 19 of 20 frames for s=20). Predictions for overlapping windows are therefore highly correlated, so per-sequence errors are not independent. The Mann-Whitney U test assumes independent observations; with positive autocorrelation, the effective sample size is far smaller than the number of windows, making the reported p<0.05 anti-conservative. The reported effect sizes are small (RMSE 0.052 vs 0.055; r 0.10 vs 0.07), so the significance stars, not the raw means, are doing the work of supporting the phrase 'significantly improved.' The paper's reported standard errors are computed across windows and inherit the same dependence. A valid comparison would need block- or cluster-level inference, for example clustering by fish or using non-overlapping windows, which the paper does not provide. Without such an analysis, the ranking 'DeepSeek-c7b is best' is not statistically established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes NLP4Neuro, a two-stage pipeline that fine-tunes off-the-shelf pre-trained transformer-based LLMs (BERT-base-uncased, GPT-2, DeepSeek Coder-7b) for sequence-to-sequence decoding of tail kinematics from whole-brain calcium imaging in larval zebrafish, and compares them with two RNN baselines (reservoir computer and LSTM). In Experiment 1, the authors report that DeepSeek-c7b yields the lowest RMSE for sequence lengths s=5, 10, and 20 and the highest Pearson r for all s, with the best RNN (LSTM) competitive at short horizons. Experiment 2 compares pre-trained versus randomly initialized transformers, Experiment 3 evaluates five input embedding strategies, and Experiment 4 applies input-gradient salience mapping to identify neurons in the pretectum as important for tail predictions. The authors conclude that pre-trained LLMs, especially mixture-of-experts models, are effective and interpretable neural decoders.","tokens_in":17330,"tokens_out":6354,"duration_ms":64573,"significance":"If the empirical results survive scrutiny, the paper offers a practically useful and reproducible demonstration that modern MoE LLMs can be fine-tuned for brain-wide neural decoding, and it provides a valuable external sanity check by validating salience maps against known pretectum-to-tail anatomy. The public code and data repositories, as well as the use of off-the-shelf model weights, make the pipeline easy to adopt. However, the headline 'DeepSeek best' rests on small absolute effect sizes (best r ≈ 0.10) and on statistical tests that do not account for the dependence between overlapping sliding windows, while the abstract's pre-training claim is internally inconsistent with the authors' own Table 2. The paper therefore establishes a promising direction rather than a definitive ranking, and the central claims require additional statistical and experimental support.","major_comments":[{"comment":"The test sequences are sliding windows of length s with stride one, so adjacent windows overlap by s−1 frames. The per-sequence RMSE values used in the Mann-Whitney U tests and the reported standard errors are therefore not independent, which inflates the effective sample size and makes the p<0.05 stars anti-conservative. Because the RMSE differences are small (e.g., 0.052 vs 0.056 at s=20), the significance asterisks, not the raw means, are doing the evidentiary work for the claim that DeepSeek-c7b is 'significantly improved.' Please provide cluster-level inference (e.g., clustering by fish or using non-overlapping windows) and report effective sample sizes.","section":"§3.4, Table 1, Fig. 2a"},{"comment":"The abstract states that 'LLMs become better at neural decoding when they use pre-trained weights learned from textual natural language data.' This claim is contradicted by Table 2: for GPT-2 at s=20 the untrained model has lower RMSE (0.052 vs 0.058) and much higher Pearson r (0.17 vs 0.04), and for DeepSeek-c7b the untrained model has higher r at both s=5 and s=20 (0.11 vs 0.10 and 0.12 vs 0.09). The text in §4.2 acknowledges the GPT-2 RMSE reversal but does not reconcile the r reversals. Please qualify the pre-training claim (e.g., to RMSE and to specific models) or re-analyze the data.","section":"Abstract, §4.2, Table 2"},{"comment":"All models are compared under a single fine-tuning protocol (learning rate 1e-6 for Experiment 1, 5–10 epochs, fixed batch sizes), and the RC and LSTM hyperparameters (reservoir size, leaking rate, input scaling, regularization, hidden size, and hidden layers) are not reported in the main text or appendix. Without evidence that the smaller baselines are reasonably tuned, the conclusion that DeepSeek-c7b is 'best' could be an artifact of under-tuned competitors. Please report the full hyperparameter settings for all models and include a small sensitivity analysis or hyperparameter search for the LSTM and RC.","section":"§3.4, Table 1"}],"minor_comments":[{"comment":"The text says 'The sum of RMSE values across all sequences is the metric reported,' but Table 1 reports values around 0.05 that must be means, not sums; please clarify whether the mean or the sum is used and define the aggregation consistently.","section":"§3.4"},{"comment":"In Eq. (14), the bias term for the second expert projection is written as b_ei^(1) in both places; the second bias should presumably be b_ei^(2).","section":"Appendix A.6, Eq. (14)"},{"comment":"The dagger symbol in Table 2 indicates a Wilcoxon signed-rank test, but the caption does not describe the test, the number of fish, or how the paired structure was defined; please describe the test and state n.","section":"Table 2"},{"comment":"The caption should explicitly state that the reservoir computer prediction is shown only in the Δθ0 panel, since the reader may otherwise expect to see it in all panels.","section":"Fig. 2c"},{"comment":"The Pt-vs-Hb salience comparison uses a Student's t-test with only three fish; please report effect sizes and consider a non-parametric or paired test, and state whether the comparison is across the three fish or across neurons.","section":"§4.4, Fig. 4c"}],"recommendation":"major_revision","confidential_remarks":"The statistical issue with overlapping windows is the most serious concern; if the authors provide cluster-robust or block-based inference and the result still holds, the headline claim would be substantially stronger. The abstract's pre-training claim is clearly overbroad relative to Table 2 and should be revised. The hyperparameter fairness issue may require additional experiments, but it is addressable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on arXiv:2507.02264. The paper asks whether off-the-shelf transformers, including a 7B mixture-of-experts model, can decode tail kinematics from whole-brain calcium imaging in larval zebrafish. The honest finding is that they can, weakly, and DeepSeek-Coder-7b looks best on raw RMSE and correlation. What's new is the systematic sweep across model classes, embedding strategies, and pre-trained vs untrained weights on a neuron-scale dataset, plus the salience mapping to known pretectum anatomy. Credit where due: code and data are public, the methods are reproducible, and the authors transparently note that GPT-2's pre-trained weights don't help at s=20.\n\nThe soft spots are serious. The significance stars in Table 1 come from Mann-Whitney U tests on per-sequence RMSE values from overlapping sliding windows. Consecutive windows share s-1 frames, so these are not independent observations; the effective sample size is close to the number of fish, not the number of windows. With effect sizes like RMSE 0.052 vs 0.055 and r=0.10 vs 0.07, the claim that DeepSeek 'significantly' outperforms the LSTM isn't established by the tests as run. The abstract overstates the pre-training benefit: Table 2 shows untrained GPT-2 beats pre-trained at s=20, and untrained DeepSeek has higher r at both sequence lengths. The authors acknowledge the GPT-2 result in the text, so it's an inconsistency between abstract and body, not a hidden result.\n\nOther weaknesses: no simple linear baseline (the reservoir computer is poor, but a ridge regression would be a more informative lower bound), and hyperparameters are fixed across models, which likely disadvantages the LSTM. The absolute accuracy is weak—best r≈0.10—so practical utility is limited to screening candidate neurons, not detailed behavioral reconstruction.\n\nOn balance, this is a useful empirical contribution that deserves a serious referee, but it's not ready as is. The authors need to fix the statistical inference (cluster by fish or use non-overlapping windows), reconcile the abstract with Table 2, add a linear baseline, and ideally report per-model hyperparameter sensitivity. The public code and data make all of this checkable.\n\nI'd bring this to a reading group to discuss the overlapping-window pitfall, and I'd cite it as an example of applying LLMs to whole-brain calcium imaging. But I'd advise the editor to require a major revision before acceptance.","headline":"Useful empirical sweep of off-the-shelf LLMs for zebrafish whole-brain decoding, but the significance tests are invalid and the abstract overstates pre-training benefits.","tokens_in":17880,"tokens_out":3667,"would_cite":true,"duration_ms":40839,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An off-the-shelf mixture-of-experts LLM, fine-tuned briefly, outperforms recurrent and other transformer models at decoding zebrafish tail movements from whole-brain calcium imaging.","keywords":["neural decoding","large language models","sequence-to-sequence learning","calcium imaging","zebrafish","mixture-of-experts","salience mapping","transfer learning"],"falsifier":"Retrain the LSTM, reservoir computer, GPT-2, and BERT baselines with per-model hyperparameter search on the same held-out fish, and also re-run DeepSeek-c7b with shuffled input tokens; if a tuned baseline matches DeepSeek's RMSE and r, or if shuffled tokens produce the same pretectal salience pattern, the paper's central claims would be undercut.","tokens_in":16867,"feed_emoji":"🧠","tokens_out":8381,"duration_ms":90607,"temperature":0.7,"pith_summary":"This paper claims that off-the-shelf large language models, fine-tuned for only a few epochs, can decode behavior from whole-brain calcium imaging better than the recurrent networks traditionally used for neural decoding. In larval zebrafish, the mixture-of-experts transformer DeepSeek-c7b produced the lowest prediction error across most sequence lengths and the highest correlation with measured tail kinematics, and its salience maps pointed to neurons in the pretectum, a region already implicated in visuomotor control. The finding matters because it suggests that text-pretrained transformer weights carry a useful inductive bias for reading neural population activity, and that a modular pipeline of open-weights LLMs could help neuroscientists identify circuit-relevant neurons for later causal tests. The absolute accuracy remains modest, so the paper's case is about relative model comparison and interpretability rather than high-fidelity decoding.","feed_headline":"DeepSeek LLM tops zebrafish neural decoding after brief tuning","feed_subtitle":"A 7-billion-parameter LLM edges out LSTMs at reading tail motion from whole-brain calcium imaging.","key_machinery":"The central mechanism is sequence-to-sequence learning with a pre-trained transformer, treated as an off-the-shelf token-level map: each time frame's whole-brain fluorescence vector is a token, and the model is fine-tuned to emit s tail predictions for s input frames. DeepSeek-c7b contributes a mixture-of-experts decoder whose routing network computes softmax scores over experts and recruits the top-k experts (k = 2) per token; the paper credits this MoE routing, plus the pre-trained textual weights, for the decoding advantage. Salience mapping then uses input-gradient scores, summed over output frames and averaged over sliding windows, to attribute each neuron's contribution to predicted tail components.","core_discovery":"On the paper's own terms, the discovery is that a 7-billion-parameter mixture-of-experts LLM, DeepSeek-c7b, fine-tuned for 5 to 10 epochs on simultaneous calcium-imaging frames and tail-angle recordings, outperforms an LSTM, a reservoir computer, GPT-2, and BERT at sequence-to-sequence prediction of tail kinematics. For sequence lengths s = 5, 10, 15, 20, DeepSeek-c7b has the lowest root-mean-square error for all but s = 15 and the best Pearson correlation with ground-truth tail sums (r about 0.08 to 0.10); it captures the timing of tail bursts better than their amplitude, and its advantage over the best RNN grows with sequence length. The paper also reports that pre-trained weights learned from natural-language text improve fine-tuned predictions for DeepSeek-c7b and BERT-bu, while relative positional encoding helps other transformers but not DeepSeek-c7b. Finally, input-gradient salience averaged over sliding windows places the most behavior-predictive neurons in the pretectum and shows lateralization consistent with the known optomotor-response circuit.","pith_inferences":["Inference: The paper does not isolate whether DeepSeek's advantage comes from the MoE architecture or from its code-heavy pretraining corpus; a same-size MoE pretrained only on general text would separate these.","Inference: The salience maps imply a specific causal experiment not run here: photostimulate the top ranked pretectal and lateralized neurons and ask whether the predicted tail direction appears.","Inference: Because best test-set r is about 0.10, the practical near-term role for NLP4Neuro is hypothesis generation about circuits rather than high-fidelity neural prostheses; benchmarking on event-triggered epochs is an immediate way to test whether frame-averaged tail sums understate its decoding ability."],"forward_implications":["Newer open-weights LLMs can be dropped into the same two-stage pipeline without redesign, so neural-decoding benchmarks can track the pace of LLM releases.","If the transfer is real, neuroscience groups without large training data can begin decoding experiments from pre-trained text models rather than training recurrent networks from scratch.","Salience maps from DeepSeek-c7b nominate pretectal and right-lateralized neurons as candidate controllers of tail behavior, giving photostimulation experiments a shortlist to test.","Longer input contexts benefit the MoE model more than the LSTM, so the approach is particularly suited to behavioral readouts that require integrating neural activity over long timescales.","Because the model captures tail event timing better than amplitude under RMSE loss, changing the loss to penalize burst amplitude may improve decoding of high-amplitude short events."],"supporting_citations":[{"why":"Supplies DeepSeek-coder-7b, the mixture-of-experts LLM whose pre-trained weights and top-k routing drive the paper's best decoding results.","marker":"[22]"},{"why":"Supplies GPT-2, the decoder-only transformer baseline that the paper compares against the MoE model.","marker":"[38]"},{"why":"Supplies BERT-base-uncased, the encoder-only transformer baseline used in Experiments 1 through 3.","marker":"[15]"},{"why":"Defines the reservoir computer, the lightweight linear-trained RNN baseline that performs worst in Experiment 1.","marker":"[19]"},{"why":"Supplies Suite2p source identification and signal extraction that produce the single-neuron fluorescence traces used as model tokens.","marker":"[36]"},{"why":"Provides CaImAn and NoRMCorre motion correction that aligns the imaging stacks before neuron extraction.","marker":"[20]"},{"why":"Documents the pretectum-to-motor optomotor circuit used to validate the anatomic consistency of DeepSeek's salience maps.","marker":"[34]"},{"why":"Supplies gradient-based salience visualization, the classical method on which the paper's input-gradient salience mapping is built.","marker":"[44]"},{"why":"Provides the transformer positional encoding scheme tested as one of five embedding strategies in Experiment 3.","marker":"[48]"}],"fun_headline_variants":["DeepSeek-7b LLM outperforms LSTMs in neural decoding","Whole-brain decoding: DeepSeek LLM edges out classical models","Pre-trained LLM bests RNNs for zebrafish tail prediction","NLP4Neuro shows LLMs excel at brain-wide behavior decoding","Mixture-of-experts LLM leads neural population decoding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that one fixed fine-tuning recipe—a learning rate of 1e-6, batch size 16, and 5 to 10 epochs—is a fair test across model families; if, say, the LSTM would beat DeepSeek-c7b with its own tuned hyperparameters, the 'best decoder' conclusion does not hold.","fun_headline_variants_meta":{"raw":{"variants":["DeepSeek-7b LLM outperforms LSTMs in neural decoding","Whole-brain decoding: DeepSeek LLM edges out classical models","Pre-trained LLM bests RNNs for zebrafish tail prediction","NLP4Neuro shows LLMs excel at brain-wide behavior decoding","Mixture-of-experts LLM leads neural population decoding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2797,"prompt_tokens":1039,"completion_tokens":1758,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":1667}},"tokens_in":655,"tokens_out":1758,"duration_ms":12722,"temperature":1.0,"reasoning_tokens":1667,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:33:22.760941+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the LSTM, reservoir computer, GPT-2, and BERT baselines with per-model hyperparameter search on the same held-out fish, and also re-run DeepSeek-c7b with shuffled input tokens; if a tuned baseline matches DeepSeek's RMSE and r, or if shuffled tokens produce the same pretectal salience pattern, the paper's central claims would be undercut.","supporting_citations":[{"cited_title":"DeepSeek-coder: When the large language model meets programming – the rise of code intelligence","cited_arxiv_id":null,"evidence_quote":"Supplies DeepSeek-coder-7b, the mixture-of-experts LLM whose pre-trained weights and top-k routing drive the paper's best decoding results."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-2, the decoder-only transformer baseline that the paper compares against the MoE model."},{"cited_title":"BERT: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Supplies BERT-base-uncased, the encoder-only transformer baseline used in Experiments 1 through 3."},{"cited_title":"Next generation reservoir computing","cited_arxiv_id":null,"evidence_quote":"Defines the reservoir computer, the lightweight linear-trained RNN baseline that performs worst in Experiment 1."},{"cited_title":"Suite2p: beyond 10,000 neurons with standard two-photon microscopy","cited_arxiv_id":null,"evidence_quote":"Supplies Suite2p source identification and signal extraction that produce the single-neuron fluorescence traces used as model tokens."},{"cited_title":"CaImAn an open source tool for scalable calcium imaging data analysis","cited_arxiv_id":null,"evidence_quote":"Provides CaImAn and NoRMCorre motion correction that aligns the imaging stacks before neuron extraction."},{"cited_title":"From whole-brain data to functional circuit models: The zebrafish optomotor response","cited_arxiv_id":null,"evidence_quote":"Documents the pretectum-to-motor optomotor circuit used to validate the anatomic consistency of DeepSeek's salience maps."},{"cited_title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","cited_arxiv_id":null,"evidence_quote":"Supplies gradient-based salience visualization, the classical method on which the paper's input-gradient salience mapping is built."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Provides the transformer positional encoding scheme tested as one of five embedding strategies in Experiment 3."}],"review_version":1}