{"id":"eb0e1991-0c97-4a2e-84bc-180a6ecbb23f","arxiv_id":"2412.11940","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Subword token granularity substantially changes language model surprisal's fit to human reading times, with an intermediate vocabulary size near 8,000 being most predictive in this paper's setup.","lead":"This paper shows that the size of the word pieces a language model uses, its token vocabulary, strongly changes how well the model's surprisal predicts human reading times. A medium vocabulary of about 8,000 pieces worked best for natural reading, a result that matters for anyone using language models to study human sentence processing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8,000-vocabulary optimum is selected on the exploratory partition without significance tests, and the held-out partition reserved for that purpose is never used; the improvement over neighboring vocabulary sizes is small and could be noise.","rationale":"The paper is careful in several respects: it includes an untrained baseline, three model sizes, eleven vocabulary sizes, multiple reading-time corpora, and publicly released models and tokenizers, and it applies a principled whitespace correction when computing word probabilities. The broad conclusion that token granularity influences surprisal's predictive power is plausible and is supported by the general shape of the results. The load-bearing vulnerability is the precision of the central claim: the 8,000-vocabulary optimum is selected by taking the maximum over exploratory-partition ΔLogLik values, and the paper itself reserves the held-out partition for significance testing but never uses it for that purpose. The margin over the next-best vocabulary size is small, and the Large model actually favors |V|=1,000, so without error bars, multiple seeds, or held-out confirmation the specific 'around 8,000' statement is not established. This concern is closely related to, but not identical with, the reader's weakest assumption about the one-epoch Mamba-2 training regime: the held-out check can be performed using the existing design, whereas the architecture/training concern requires additional training runs. The CONDITIONAL verdict remains appropriate because the issue is addressable and the broad effect is believable; only the precise optimum needs stronger support.","tokens_in":14300,"tokens_out":5466,"duration_ms":53567,"concrete_test":"Recompute Experiment 1's ΔLogLik on the held-out partition, using LME models fit on the fit partition, for the top three vocabulary sizes (|V|=4,000, 8,000, 16,000) across all three model sizes and all corpora, and compare them with paired bootstrap/permutation confidence intervals over subjects and items. In addition, train 2–3 seeds of the Small and Medium models at |V|=4,000, 8,000, and 16,000 and recompute Figure 2. If |V|=8,000 is not the highest, or is within the confidence interval of 4,000 or 16,000 on held-out data, the conclusion should be softened to 'intermediate vocabularies around 4k–16k are competitive,' with the exploratory selection explicitly noted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim of an 8,000-token optimum rests on exploratory-partition ΔLogLik values with no uncertainty quantification. In Section 3.3 the authors partition each corpus into fit, exploratory, and held-out partitions and state that the held-out partition is reserved for statistical significance testing, but Section 3.4 reports only aggregated exploratory ΔLogLik (Figure 2; Table 5). The advantage of |V|=8,000 over |V|=1,000 is small: across the three model sizes the summed totals are about 11,334 vs. 11,278, and the Large models show the opposite ordering (|V|=1,000: 3,643.2; |V|=8,000: 3,606.9). Because eleven vocabulary sizes are compared without multiple seeds, error bars, or significance tests, selecting the maximum on the exploratory partition can easily reflect chance rather than a stable granularity optimum. This matters because the abstract and Section 3.4 make the specific quantitative claim that vocabulary sizes 'around 8,000' yield the strongest surprisal predictors, not merely that intermediate granularities are competitive.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how subword token granularity (vocabulary size) affects the ability of language model surprisal to predict human reading times and to estimate garden-path effects. The authors train ULM tokenizers with 11 vocabulary sizes and Mamba-2 language models in three sizes, evaluate surprisal on 10 reading-time measures from five corpora, and on a garden-path benchmark following Huang et al. (2024). They report that an intermediate vocabulary size around 8,000 yields the best fit to naturalistic reading times (averaged over model sizes), and that coarser-grained tokens generally lead to larger surprisal differences at garden-path critical regions. They also compare with GPT-2 Small and provide untrained-model baselines to separate initial tokenization bias from learned representations.","tokens_in":14439,"tokens_out":7495,"duration_ms":59054,"significance":"If the findings are robust, the paper identifies a previously overlooked modeling decision—subword vocabulary size—as a first-order variable in cognitive modeling of reading. The experimental scope is substantial: 33 trained models, 11 vocabulary sizes, 10 reading-time datasets, and an external garden-path benchmark. The untrained-versus-trained comparison is a clean design for disentangling initial tokenization bias from learned representations, and the authors publicly release their tokenizers and models (a strength for reproducibility). However, in its current form the inferential support for the headline optimum is weak, and the garden-path claim is not consistently supported by the estimated effects.","major_comments":[{"comment":"The claim that 'a vocabulary size of around 8,000' yields the strongest predictors (Abstract; Section 3.4) is selected and reported on the same exploratory partition, and the held-out partition reserved for significance testing is never used. The evidence for 8,000 over neighboring sizes is small and inconsistent: summing the totals in Table 5 over the three model sizes gives 11,333.7 for |V|=8,000 versus 11,277.8 for |V|=1,000, and the Large models show the opposite ordering (3,606.9 vs. 3,643.2). Because eleven sizes are compared without confidence intervals, multiple-seed variability, or any significance test, the specific optimum is not established. The authors should either perform confirmatory tests on the held-out partition or substantially temper the headline claim to an 'intermediate granularity' trend.","section":"Section 3.3 and Section 3.4"},{"comment":"The abstract's conclusion that 'LMs trained on coarser-grained tokens generally assigned higher surprisal to critical regions ... suggesting a greater sensitivity to garden-path effects' is not supported by the estimated GPEs in Figure 3, which show no consistent monotonic trend: the Medium LM peaks at |V|=1,000, and the Large LMs are flat or noisy. The raw surprisal differences in Figure 4 do show a coarse-token trend, but the paper itself acknowledges that the estimated GPE is influenced by the linking function. The claim of greater GPE sensitivity should be restricted to raw surprisal, or the discrepancy with the estimated effects should be addressed.","section":"Section 4.2, Figures 3 and 4"},{"comment":"The averaging across model sizes masks large heterogeneity in the post-training optimum: the Small LM peaks at |V|=8,000, but the Medium and Large LMs peak at |V|=1,000. Moreover, the untrained peak in Table 3 is at |V|=4,000 for all sizes, so the statement in Section 3.4 that Small LMs 'replicate the peak observed prior to LM training' is inaccurate—the Small LM's trained peak shifts from 4,000 to 8,000. The conclusion that 8,000 is the optimal granularity after training is therefore an artifact of averaging, and the paper should either present the model-size-specific patterns as the main result or justify the pooling.","section":"Table 5 and Figure 2"}],"minor_comments":[{"comment":"The phrase '24 items of the Main Verb/Reduced Relative (MV/RR), Direct Object/Sentential Complement (NP/S), and Transitive/Intransitive (NP/Z) garden-path constructions' is ambiguous—please clarify whether there are 24 items per construction or 24 items total.","section":"Section 4.1"},{"comment":"The comparison with GPT-2 Small is not matched on training data or model size; the paper should note that this is a reference point, not a controlled comparison, or provide a matched baseline.","section":"Section 3.4"},{"comment":"The limitations section does not mention the use of a single architecture (Mamba-2), a single training epoch, and no multiple random seeds; adding a sentence on these points would help calibrate the generalizability of the conclusions.","section":"Limitations"},{"comment":"The abstract states 'a vocabulary size of 8,000' while Section 3.4 says 'around 8,000'; align these statements to avoid overstating the precision of the finding.","section":"Abstract and Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the experimental design is thoughtful, with a commendable release of code and models. The main issue is that the headline claim needs inferential support; this is fixable by analyzing the already-reserved held-out partition or by adding uncertainty quantification. The garden-path claim also needs to be reconciled with the estimated effects. I recommend a major revision focused on these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is the first systematic sweep of subword vocabulary size for surprisal-based cognitive modeling, and the untrained-vs-trained comparison is a genuinely nice design. The broad conclusion—token granularity is a real methodological variable—holds up. The specific 8k optimum does not, as the paper reports it.\n\nWhat's new: 11 ULM vocabulary sizes, three Mamba-2 sizes, ten reading-time datasets, plus a garden-path analysis following Huang et al. The untrained baseline shows granularity alone predicts reading times, peaking around 4k-16k, and that initial bias partly persists in small models. That's a useful empirical result and a fair contribution.\n\nSoft spots, in order. First, the headline optimum is selected and reported on the exploratory partition; the held-out partition, explicitly reserved for significance testing, is never used. Differences between 8k and nearby vocabularies are small (aggregate ΔLogLik deltas around 100-300 across hundreds of thousands of points), and the Large models actually favor 1k over 8k. With one seed per condition and no error bars, the precise 8k claim is not supported. The paper should either do the held-out test or soften the claim to \"intermediate granularities are competitive.\" Second, the garden-path experiment estimates GPE from model-predicted reading times, not human data, and the trend is noisy; the raw surprisal difference at the critical word is the more direct measure and does show a coarse-token trend, but it's qualified. Third, the single-epoch Mamba-2 training is a narrow regime; whether the interaction survives with fully trained Transformers is open.\n\nThe authors are honest about limitations (English only, model sizes). The citation pattern is fair, and the code and models are released. This paper deserves a serious referee: the question is important, the data collection is substantial, and the core finding is probably right even if the specific optimum is overclaimed. I would send it to review, with a request to either do the held-out significance test or reframe the conclusion.","headline":"Systematic sweep of subword granularity shows tokenization matters for surprisal-based cognitive modeling, but the specific 8k optimum is not statistically supported.","tokens_in":15073,"tokens_out":1590,"would_cite":true,"duration_ms":14516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that subword token granularity is a first-order methodological variable in cognitive modeling: a vocabulary size around 8,000 makes surprisal the strongest predictor of naturalistic reading times, while coarser tokens…","keywords":["surprisal","token granularity","subword tokenization","unigram language model","reading times","garden-path effects","state-space language models","cognitive modeling"],"falsifier":"A decisive check would be to retrain all eleven vocabulary conditions with several random seeds and more than one epoch, then test whether the roughly 8,000-token optimum in aggregate log-likelihood gain and the coarser-token garden-path trend still appear; if the optimum shifts or vanishes across seeds, the peak is an artifact of a single training run.","tokens_in":14000,"feed_emoji":"📖","tokens_out":10175,"duration_ms":81292,"temperature":0.7,"pith_summary":"This paper asks whether the choice of subword token granularity changes how well language-model surprisal, the negative log-probability a model assigns to a word, predicts human reading behavior. It trains unigram-tokenized Mamba-2 language models of three sizes with vocabularies from 256 to 128,000 tokens, then tests surprisal against ten measures from five reading-time corpora and against garden-path constructions. The paper concludes that token granularity matters substantially: before and after training, an intermediate vocabulary around 8,000 tokens produces surprisal that best fits naturalistic reading times, exceeding a widely used reference model. On garden-path sentences, coarser, more word-like tokens generally assign higher surprisal to critical regions, implying larger garden-path effects than previously reported. If correct, token granularity should be treated as a methodological variable rather than a neutral preprocessing choice.","feed_headline":"8,000-token vocab best predicts human reading times","feed_subtitle":"Study shows subword token granularity shifts both surprisal fit to naturalistic text and garden-path effects.","key_machinery":"The central object is token granularity, operationalized by ULM, a unigram-language-model tokenizer that prunes a large character-based inventory down to a target vocabulary size. These tokenizers feed autoregressive Mamba-2 state-space language models in three sizes, and word probabilities are recovered through a leading-whitespace correction. Surprisal's predictive power is scored as the increase in regression model log-likelihood from adding surprisal to linear mixed-effects models of reading times, and garden-path effects are estimated by fitting a surprisal-to-reading-time linking function on filler items and comparing predicted reading times across ambiguous and control conditions.","core_discovery":"On the paper's own terms, the discovery is that the subword vocabulary size is itself a cognitive-modeling parameter. Finer granularities make word surprisal track word length and character-level predictability, while coarser granularities make tokens increasingly word-like so that word surprisal approaches uniformity across common words and loses predictive power. The paper finds a sweet spot at roughly 8,000-token vocabularies on average across model sizes after training, with smaller models preserving an earlier peak near 4,000 tokens before training. In the garden-path experiment, coarser-grained tokens generally give larger differences in surprisal between ambiguous and unambiguous conditions at critical words, while still underestimating human garden-path effects by one or two orders of magnitude.","pith_inferences":["An obvious extension is to repeat the vocabulary sweep on non-English corpora; because ULM uses characters as base units, writing systems with deeper orthography could shift the optimal vocabulary size away from 8,000.","The near-optimality of untrained 4,000-token surprisal suggests a cheap diagnostic: a researcher could select a tokenizer by reading-time fit on a small sample before committing to expensive language-model training.","If the size interaction generalizes, the practical recommendation is not a single best vocabulary but a size matched to model scale, with larger models tolerating coarser tokens.","A concrete test would be to hold the vocabulary at 8,000 and vary model size beyond the Large model used here; if the granularity peak flattens further, the cognitive relevance of token granularity may be a small-model phenomenon."],"forward_implications":["Vocabulary size alone can change a model's reading-time fit by hundreds of log-likelihood points, so studies using different tokenizers are not directly comparable.","The conventional vocabulary size around 50,000 is not optimal for naturalistic comprehension; models trained with roughly 8,000 tokens fit aggregate reading times better than the GPT-2 Small reference.","Coarser token granularity increases the raw surprisal difference between ambiguous and unambiguous garden-path sentences, which means tokenizer choice can inflate or deflate estimated garden-path effects.","The effect weakens as model size grows, so scaling may partially absorb the initial bias imposed by tokenization."],"supporting_citations":[{"why":"provides the unigram-language-model tokenizer used to construct the 11 vocabulary-size conditions.","marker":"Kudo, 2018"},{"why":"provides the Mamba-2 state-space architecture and multi-input setup used for all trained language models.","marker":"Dao and Gu, 2024"},{"why":"supplies the Wiki-40B English text used to train the tokenizers and the language models.","marker":"Guo et al., 2020"},{"why":"supplies the Natural Stories self-paced reading corpus used in Experiment 1.","marker":"Futrell et al., 2021"},{"why":"supplies the Brown self-paced reading corpus used in Experiment 1.","marker":"Smith and Levy, 2013"},{"why":"supplies the GECO eye-tracking corpus used in Experiment 1.","marker":"Cop et al., 2017"},{"why":"supplies the Dundee eye-tracking corpus used in Experiment 1.","marker":"Kennedy et al., 2003"},{"why":"supplies the Provo corpus used in Experiment 1 and the filler items for the garden-path reading-time model.","marker":"Luke and Christianson, 2018"},{"why":"supplies the garden-path stimuli, reading-time data, and estimation procedure used in Experiment 2.","marker":"Huang et al., 2024"},{"why":"supplies the leading-whitespace correction used to convert token probabilities to word probabilities.","marker":"Oh and Schuler, 2024"}],"fun_headline_variants":["8K-token vocab best for predicting human reading times","Token size shifts surprisal's power to explain reading difficulty","Coarser tokens heighten garden-path surprisal gaps in LLMs","Subword vocabulary size is a key factor for surprisal models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Mamba-2 models trained for a single pass over Wiki-40B, with no repeated random initializations, stand in for language models generally; if longer training, a different architecture, or another seed changes how token granularity interacts with learning, the observed 8,000-token optimum and the garden-path granularity trend could be specific to this training regime.","fun_headline_variants_meta":{"raw":{"variants":["8K-token vocab best for predicting human reading times","Token size shifts surprisal's power to explain reading difficulty","Coarser tokens heighten garden-path surprisal gaps in LLMs","Subword vocabulary size is a key factor for surprisal models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1437,"prompt_tokens":882,"completion_tokens":555,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":482}},"tokens_in":498,"tokens_out":555,"duration_ms":5462,"temperature":1.0,"reasoning_tokens":482,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:26:00.941307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would be to retrain all eleven vocabulary conditions with several random seeds and more than one epoch, then test whether the roughly 8,000-token optimum in aggregate log-likelihood gain and the coarser-token garden-path trend still appear; if the optimum shifts or vanishes across seeds, the peak is an artifact of a single training run.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the unigram-language-model tokenizer used to construct the 11 vocabulary-size conditions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the Mamba-2 state-space architecture and multi-input setup used for all trained language models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Wiki-40B English text used to train the tokenizers and the language models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the garden-path stimuli, reading-time data, and estimation procedure used in Experiment 2."}],"review_version":1}