{"id":"5c064916-9774-4198-852b-4fd98871fd43","arxiv_id":"2502.00344","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A transformer language model trained on tokenized Bengalese finch songs outperforms Markov, RNN, and LSTM baselines and provides evidence for long-range syllable dependencies.","lead":"The authors trained a GPT-style language model on text versions of Bengalese finch songs and found it predicts the next syllable better than Markov, RNN, and LSTM models. The model's attention patterns and experiments that limit how much past context it can use suggest finch songs contain dependencies that reach more than six syllables back.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 6th-order Markov null corpus is not shown to match natural song-length and boundary statistics, so the Fig. 1E accuracy drop does not uniquely establish long-range dependencies beyond six syllables.","rationale":"The reader's verdict is CONDITIONAL, and the reader identifies the faithfulness of the artificial 6th-order Markov corpus as the weakest assumption. My stress test agrees that this is the load-bearing point. The paper provides no code, data, or diagnostics on the null corpus, so the concern cannot be dismissed from the text. The attention-span restriction results in Fig. 2D and Fig. 4E-F provide convergent evidence that long attention context matters, but they do not isolate dependencies beyond six syllables; they show sensitivity to truncating context and could reflect lower-order chunk boundaries or distribution shift. Therefore the central claim is exactly as secure as the null corpus. Re-running the comparison with a length-matched artificial corpus is a tractable and decisive check. I do not see an additional objection that would change the reader's conditional verdict; the appropriate outcome is to keep the verdict CONDITIONAL pending that check, so no change to the reader's verdict is needed.","tokens_in":12733,"tokens_out":3467,"duration_ms":38630,"concrete_test":"Regenerate the artificial corpus with exactly the natural corpus's song-length distribution: for each natural song of length L, sample a length-L sequence from the estimated 6th-order transition matrix (using the appropriate start context and stopping at L rather than sampling until an end token), then retrain and evaluate FinchGPT and the 6th-order Markov baseline on this length-matched corpus. If FinchGPT's accuracy drop relative to the natural corpus disappears or shrinks to nonsignificance, the Fig. 1E result is an artifact of length or boundary mismatch rather than evidence for dependencies beyond six syllables. Also report and compare the length distributions, per-position accuracy, and start/end-token frequencies for the natural and artificial corpora.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core inference that Bengalese finch songs contain functionally important dependencies beyond six syllables rests on Fig. 1E: FinchGPT's accuracy drops on an artificial corpus generated from a 6th-order Markov model, while the Markov baseline does not. The Methods section states only that transition probabilities including start and end tokens were estimated and that songs were generated by iterating until an end token was emitted. It does not state that the generated corpus matches the natural corpus in song-length distribution, in the frequency of start/end contexts, or in any statistics beyond the modeled order. Sampling until an end token imposes a length distribution determined by the estimated transition probabilities, which can differ substantially from the empirical length distribution. Because next-token accuracy is sensitive to sequence length and boundary placement, a mismatch in these incidental statistics could lower FinchGPT's accuracy on the artificial corpus even if the dependency structure beyond six syllables is identical. The paper reports no diagnostic comparing natural and artificial corpora on length distribution, per-position accuracy, or entropy, so the contrast in Fig. 1E is confounded: the absence of long-range dependencies is not the only difference between the two test sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinchGPT, a GPT-2-style transformer trained from scratch on texturized Bengalese finch songs from three individual birds. It reports that this model achieves higher next-syllable prediction accuracy and lower cross-entropy than Markov, RNN, and LSTM baselines (Fig. 1C–D), and that its accuracy drops more than the Markov baseline when both are evaluated on an artificial corpus generated from a 6th-order Markov model (Fig. 1E). Attention visualization is used to argue that deeper layers attend to longer spans (Fig. 2A–C), and restricting the attention span degrades performance (Fig. 2D). Finally, the authors show that HVC ablation changes song statistics and reduces FinchGPT's performance and classification accuracy, which they interpret as a loss of long-range dependencies (Fig. 4).","tokens_in":12942,"tokens_out":8223,"duration_ms":78988,"significance":"The paper asks a timely and important question: whether transformer language models can reveal non-adjacent sequential structure in animal vocalizations. Its strengths include a multi-bird corpus, a causal attention-span manipulation, and the use of a Markov null corpus as a reference point. If the central claim is established, the approach could provide a broadly applicable tool for comparative studies of vocal sequence structure. However, the current evidence is not yet sufficient: the Markov null corpus is not shown to match the natural corpus in incidental statistics, and the main accuracy and scaling comparisons lack reported inferential statistics. These issues are fixable with additional analyses.","major_comments":[{"comment":"The artificial 6th-order Markov corpus is not demonstrated to match the natural corpus on song-length distribution and boundary statistics. The Methods state only that transition probabilities including start and end tokens were estimated and that songs were generated by iterating until an end token was emitted. This sampling scheme imposes the length distribution implied by the estimated transition probabilities, which need not equal the empirical length distribution; next-token accuracy is sensitive to sequence length and boundary placement. Consequently, the accuracy drop in Fig. 1E is confounded: absence of dependencies beyond six syllables is not the only difference between the natural and artificial test sets. Please report side-by-side diagnostics (song-length distribution, start/end token frequencies, per-position accuracy, unigram and bigram frequencies) and, if mismatches appear, re-run the comparison on a length-matched artificial corpus or include a scrambled-order control.","section":"§3, 'Procedures for training Markov, RNN, LSTM, and Transformer models'; Fig. 1E"},{"comment":"The claim that FinchGPT outperforms Markov, RNN, and LSTM models at next-syllable prediction is central but lacks inferential statistics. The Statistics section lists paired t-test and Wilcoxon tests, yet the Results report p-values only for Fig. 1E (p = 0.033 and p = 0.31). With n = 3 corpora, the accuracy and cross-entropy differences in Fig. 1C and 1D must be accompanied by p-values or confidence intervals for each pairwise comparison. The same applies to the 'significantly worse' claim for the 1L/1A model in Fig. 1G and to the data-size comparisons in Fig. 1H.","section":"§4.1, Fig. 1C–D"},{"comment":"The restricted-attention experiment is a strong causal design, but the functional-importance conclusion needs statistical support. No p-value or confidence interval is reported for the increase in cross-entropy as the attention span is restricted, despite the caption giving n = 235,256 predictions across 2,660 songs. Because masking changes the model's effective context and its training dynamics, please also report a control that separates context-length effects from capacity or optimization effects, such as a full-attention model with comparable effective capacity or a fixed-context Markov baseline evaluated under the same masking.","section":"§4.3, Fig. 2D"},{"comment":"The before/after HVC-ablation comparison is potentially confounded by distribution shift. The after-ablation corpus has 1,180 songs versus 752 before ablation, and 59,111 versus 38,599 predictions; the paper reports that pitch is unchanged but does not report whether syllable frequencies or song lengths changed after ablation. The increased cross-entropy of the before-trained model on after-ablation songs could reflect marginal distribution shift rather than a specific loss of long-range dependencies. Please provide these diagnostics, report the p-values for the Wilcoxon tests in Fig. 4D, and add error bars or statistics for the attention-span comparisons in Fig. 4E–F.","section":"§4.5, Fig. 4D–F"}],"minor_comments":[{"comment":"The attention-weight threshold of >0.5 used to define 'attention span length' is arbitrary; please justify it or provide a sensitivity analysis over thresholds.","section":"Fig. 2C"},{"comment":"Excluding the token immediately following the start token from the cross-entropy calculation should be justified, since this choice changes the metric and affects comparability across models and corpora.","section":"Methods, 'Model Evaluation Indicators'"},{"comment":"The manuscript does not include a data or code availability statement, which is important for reproducing the corpus, the artificial-corpus generation, and the model training.","section":"Throughout"},{"comment":"There are several typographical and labeling issues: 'attenuation heads' should read 'attention heads' (Methods); 'aniterior' in Fig. 4A should be 'anterior'; and 'Rate of change' in the Fig. 4F caption should read 'ROC'.","section":"Editorial"},{"comment":"The sentence 'The model engineered one attention head layer (1L, 1A) resulted in significantly worse cross-entropy' is awkward and should be rewritten for clarity.","section":"§4.2, Fig. 1G"},{"comment":"The caption states that the upper bound is the accuracy achieved when testing on the training dataset; this is an unusual definition of an upper bound and should be explained more clearly in the text.","section":"Fig. 4F"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid cross-disciplinary empirical study, but the central long-range-dependency inference currently rests on a confounded control and incomplete statistics. The authors should be asked to provide the corpus diagnostics and significance tests described in the major comments. I would also encourage the editor to require a data and code availability statement, since the artificial-corpus generation is central to the argument and otherwise cannot be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nFinchGPT is the first from-scratch transformer LM on tokenized birdsong, and the attention-span restriction is a genuinely nice causal probe. The paper also connects computational manipulations to a real biological one (HVC ablation) and shows a coherent story: the model beats Markov/RNN/LSTM at next-syllable prediction, attention reaches beyond local motifs, and restricting attention hurts performance. The embedding analysis showing context-dependent syllable usage is a reasonable bonus.\n\nBut the load-bearing evidence for long-range dependencies beyond six syllables is Fig 1E, and that comparison has a confound. The artificial corpus is generated by sampling a 6th-order Markov model until an end token. The Methods never show that the generated songs match the natural corpus in length distribution, boundary statistics, or anything else not captured by the modeled order. Next-token accuracy is sensitive to position and sequence length, so a mismatch in those incidental statistics could lower the transformer's accuracy on the artificial corpus even if the dependency structure were the same. The paper reports no diagnostic comparing the two corpora on length, entropy, or per-position accuracy. This is not a minor quibble; it's the main evidence for the 'beyond six syllables' claim.\n\nOther soft spots are smaller. The headline comparisons in Fig 1C/D lack significance tests (only Fig 1E has p-values). AUC values in Fig 4F have no error bars. The RNN/LSTM were tuned with TPE but the transformer was not, so part of the performance gap could be tuning asymmetry. No data or code is provided, which makes these checks hard for readers. The citation pattern is appropriate, covering relevant birdsong and NLP work.\n\nI still think the paper is worth serious engagement. The application is novel, the attention restriction idea is good, and the HVC story is interesting. But the long-range claim needs a cleaner null corpus or additional diagnostics. If the authors fix that, it would be a solid contribution to bioacoustics and comparative cognition.","headline":"A novel and promising application of transformers to birdsong, but the key evidence for long-range dependencies rests on an uncontrolled synthetic corpus.","tokens_in":13476,"tokens_out":3155,"would_cite":false,"duration_ms":28813,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A transformer trained from scratch on Bengalese finch songs predicts the next syllable better than Markov, RNN, and LSTM models, and its attention maps reveal long-range dependencies in the song structure.","keywords":["birdsong","Bengalese finch","transformer language model","next-syllable prediction","long-range dependencies","attention analysis","HVC ablation","animal vocalization"],"falsifier":"Regenerate the artificial corpus from the 6th-order Markov model while explicitly matching the natural corpus' song-length distribution and start/end token handling, then retrain FinchGPT on it. If the transformer's accuracy no longer drops relative to the Markov model, the reported decline is an artifact of the generation procedure, not evidence that natural songs contain dependencies beyond six syllables.","tokens_in":12550,"feed_emoji":"🐦","tokens_out":11514,"duration_ms":107111,"temperature":0.7,"pith_summary":"The paper claims that a GPT-2-style transformer, trained from scratch on text-converted Bengalese finch songs, predicts the next syllable more accurately than Markov, RNN, and LSTM models. It argues the transformer's advantage comes from long-range dependencies in the songs, not merely local transition statistics. Evidence includes attention visualization showing distant syllables attended to in deeper layers, and experiments where restricting the attention span to a few syllables hurts prediction. The paper also shows that songs from birds after HVC ablation lose the long-range structure the model relies on, linking the model's behavior to a biological manipulation of song syntax. If correct, transformer language models become a general tool for detecting hierarchical structure in non-human vocalizations.","feed_headline":"FinchGPT beats Markov and LSTM at next-syllable prediction","feed_subtitle":"A GPT trained on finch syllables exposes rules a six-syllable Markov window misses.","key_machinery":"The load-bearing object is the self-attention mechanism of a GPT-2-style transformer applied to a tokenized song corpus, with each syllable as a token and start and end markers. The model is compared against a 6th-order Markov model on natural songs and on a synthetic corpus with the same local statistics but no dependencies beyond six syllables; that synthetic corpus is the null model that isolates the contribution of long-range context. Attention span restriction, implemented by masking tokens farther away than a fixed distance in every layer, is the experiment that turns the architecture's access to distant context on and off. The paper also uses a graph-theoretic extraction of dependency trees from attention weights to visualize which syllables depend on which, and HVC ablation as a biological manipulation that disrupts long-range ordering without changing syllable acoustics.","core_discovery":"The central discovery is that Bengalese finch songs contain predictive dependencies spanning more than six syllables, and a transformer can exploit them. FinchGPT, a GPT-2-style model with 6 layers, 6 attention heads, and 384 hidden units, outperforms a 6th-order Markov model and recurrent networks at next-syllable prediction on corpora from three birds. When the same architecture is trained and tested on an artificial corpus generated by a 6th-order Markov process that by construction contains no dependencies beyond six syllables, the transformer's accuracy drops significantly while Markov's does not. Attention weights in later layers reach across motifs, and limiting the attention span to fewer than roughly six preceding tokens monotonically worsens cross-entropy. The paper interprets these results as evidence that the songs are not Markovian and that long-range sequential rules are functionally relevant.","pith_inferences":["A testable extension would be to apply the same attention-span-restriction protocol to recordings from other species; the distance at which cross-entropy rises would give a comparable context length for each species' vocal sequences.","A direct comparison with biology would record HVC neurons during singing and ask whether their sequence selectivity matches FinchGPT's effective receptive field, turning the model's attention maps into hypotheses about neural coding.","The synthetic Markov null is one possible baseline; a more demanding null would be a hierarchical grammar with variable motif repetition, which could reveal whether the transformer's advantage is detecting genuine long-range syntax rather than statistical regularities in chunk boundaries."],"forward_implications":["Next-syllable accuracy and cross-entropy on song corpora can rank model architectures, with attention-based models capturing structure that finite-order Markov and recurrent models miss.","Restricting attention span provides a quantitative readout of the distance over which sequential dependencies operate; for Bengalese finches, the steepest degradation appears between 3 and 10 syllables.","HVC ablation removes long-range syllabic order while leaving syllable acoustics intact, so the same model can detect loss of syntactic structure from song text alone.","A transformer trained on pre-ablation songs can classify whether a song was produced before or after HVC ablation, with accuracy falling as attention span shrinks."],"supporting_citations":[{"why":"Supplies the Transformer self-attention architecture on which FinchGPT is built.","marker":"[1]"},{"why":"Defines the GPT-2-style decoder architecture used for FinchGPT.","marker":"[2]"},{"why":"Provides the automated syllable annotator SAIBS that converts recorded songs into the tokenized text corpus.","marker":"[13]"},{"why":"Establishes that animal vocal sequences are not Markov chains, motivating the comparison against Markov baselines.","marker":"[10]"},{"why":"Documents long-range order in canary song, the phenomenon the paper extends to Bengalese finches.","marker":"[11]"},{"why":"Provides evidence of parallels in the sequential organization of birdsong and human speech, supporting the interpretability target.","marker":"[27]"},{"why":"Supplies the graph algorithm used to extract dependency parses from attention weights in the visualization analysis.","marker":"[24, 25]"}],"fun_headline_variants":["Transformer unravels hidden long-range rules in finch songs","Birdsong has grammar beyond six syllables, GPT shows","FinchGPT finds bird song rules that Markov models miss","FinchGPT exposes long-range structure in bird vocalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument for long-range structure depends on the artificial 6th-order Markov corpus faithfully removing all dependencies beyond six syllables while preserving every other statistical property of the natural songs, since any artifact in that generation procedure could produce the transformer's accuracy drop without genuine long-range dependencies.","fun_headline_variants_meta":{"raw":{"variants":["Transformer unravels hidden long-range rules in finch songs","Birdsong has grammar beyond six syllables, GPT shows","FinchGPT finds bird song rules that Markov models miss","FinchGPT exposes long-range structure in bird vocalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3485,"prompt_tokens":932,"completion_tokens":2553,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2486}},"tokens_in":548,"tokens_out":2553,"duration_ms":19006,"temperature":1.0,"reasoning_tokens":2486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:21:38.658874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the artificial corpus from the 6th-order Markov model while explicitly matching the natural corpus' song-length distribution and start/end token handling, then retrain FinchGPT on it. If the transformer's accuracy no longer drops relative to the Markov model, the reported decline is an artifact of the generation procedure, not evidence that natural songs contain dependencies beyond six syllables.","supporting_citations":[{"cited_title":"Vaswani, et al., Attention is All you Need in Advances in Neural Information Processing Systems, (Curran Associates, Inc., 2017)","cited_arxiv_id":null,"evidence_quote":"Supplies the Transformer self-attention architecture on which FinchGPT is built."},{"cited_title":"Radford, et al., Language Models are Unsupervised Multitask Learners","cited_arxiv_id":null,"evidence_quote":"Defines the GPT-2-style decoder architecture used for FinchGPT."},{"cited_title":"Kawaji, M","cited_arxiv_id":null,"evidence_quote":"Provides the automated syllable annotator SAIBS that converts recorded songs into the tokenized text corpus."},{"cited_title":"Kershenbaum, et al., Animal vocal sequences: not the Markov chains we thought they were","cited_arxiv_id":null,"evidence_quote":"Establishes that animal vocal sequences are not Markov chains, motivating the comparison against Markov baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents long-range order in canary song, the phenomenon the paper extends to Bengalese finches."},{"cited_title":"Sainburg, B","cited_arxiv_id":null,"evidence_quote":"Provides evidence of parallels in the sequential organization of birdsong and human speech, supporting the interpretability target."}],"review_version":1}