{"id":"0b9ceabb-8fa4-433f-8d8d-4e9f62435a3d","arxiv_id":"2507.22209","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Monte Carlo estimates of whole-word entropy diverge from first-token entropy estimates and often fit reading times better, suggesting the first-token approximation is unreliable for psycholinguistic modeling.","lead":"This paper compares two ways of measuring how unpredictable a word is in context: the common shortcut of looking only at the first subword token, and a Monte Carlo method that considers whole words. It finds the two measures produce divergent predictions for reading times, so results from the shortcut should be treated with caution.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's per-corpus ΔLL values lack uncertainty intervals and the only significance test pools heterogeneous corpora, so the claimed 'concrete difference' behind the central caution is not yet statistically established.","rationale":"The paper's contribution is an empirical warning: first-token entropy and MC word entropy lead to divergent psycholinguistic conclusions. That warning lives or dies on the statistical comparison in Table 2. The MC sampling procedure follows Oh and Schuler (2024) and includes a bootstrap variance analysis in Appendix A.2, so I do not treat the sampling method itself as the weakest link; the missing uncertainty quantification is more direct. The authors report ΔLL aggregates without intervals, and the only significance test pools squared errors across corpora with different response scales. Because many individual ΔLL values are near zero, the aggregate p-values could be driven by a subset of large effects. A good-faith skeptical reader needs per-corpus intervals or per-corpus permutation tests to see whether the 'concrete difference' is widespread. This concern does not refute the paper; it sets the evidentiary bar for the central empirical claim. The reader's CONDITIONAL verdict remains appropriate, and requiring the additional statistical reporting would address the concern directly.","tokens_in":7713,"tokens_out":8154,"duration_ms":102387,"concrete_test":"For each row of Table 2, compute bootstrap or profile-likelihood confidence intervals for ΔLL by refitting the LME models on resampled held-out partitions, and run the permutation test separately for each corpus and response type before pooling. If most per-corpus intervals overlap zero or point in mixed directions, the 'concrete difference' claim should be weakened; if the intervals separate first-token from MC in the reported direction for the majority of corpora, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion in Sec. 5—that the concrete difference between first-token and MC entropy 'warrants caution'—rests on Table 2 and on a single permutation test over squared errors aggregated across all reading-time corpora. The sixteen ΔLL point estimates in Table 2 are reported without standard errors, profile-likelihood intervals, or per-corpus significance tests. Several entries are close to zero or negative (e.g., Provo FP Shannon first-token −0.2; GECO FP MC Shannon −0.4), so the sign of an individual comparison is uninformative without uncertainty. Aggregating squared errors across self-paced reading times, first-pass durations, and go-past durations mixes different units and noise levels; one or two large effects (e.g., Natural Stories SPR Shannon 29 vs. 72) could drive the reported p < 0.01 / p < 0.001 even if most individual comparisons were null. The paper's claim is specifically about divergence in psycholinguistic predictions, so the absence of per-corpus uncertainty quantification is the load-bearing gap. This is not a criticism of the MC sampling framework per se; it is a question of whether Table 2 actually establishes the divergence the Discussion asserts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares first-token (subword-level) approximations of contextual Shannon and Rényi entropy with Monte Carlo estimates that allow a word to span multiple subword tokens, using GPT2-small as the language model. The two entropy estimators are evaluated as predictors of reading times in two self-paced reading corpora and three eye-tracking corpora, with surprisal and other baseline predictors controlled in linear mixed-effects regressions. The paper reports that first-token entropy is a lower bound on word entropy, demonstrates in Figure 1 that MC entropy is on average higher and differs across parts of speech, and reports in Table 2 that the two estimators yield different log-likelihood improvements. It concludes that the concrete differences between first-token and MC entropy warrant caution against using first-token entropy in psycholinguistic modeling.","tokens_in":7887,"tokens_out":6458,"duration_ms":87174,"significance":"If the reported divergence is real, this is a useful methodological caution for a research community that frequently uses first-token entropy as a proxy for word-level contextual entropy. The paper has several concrete strengths: the code is released, the connection to the whitespace-initial-token confound is explicitly acknowledged with references to Oh and Schuler (2024) and Pimentel and Meister (2024), the MC sampling procedure is described in enough detail to be reproduced, and Appendix A.2 provides a bootstrap-based variance analysis of sample count. The theoretical lower-bound point is not new, so the paper's contribution is primarily empirical. The main weakness is that the central empirical claim is supported by point estimates and an aggregated permutation test rather than by per-corpus uncertainty quantification, which is the load-bearing gap for the Discussion's conclusion.","major_comments":[{"comment":"The claim that first-token and MC entropy yield a 'concrete difference' rests on ΔLL point estimates in Table 2 that have no standard errors, confidence intervals, or per-corpus significance tests. Several entries are close to zero or negative (e.g., Provo FP Shannon first-token −0.2; GECO FP MC Shannon −0.4), so the sign of an individual comparison is not informative without uncertainty. The permutation test described in §4.3 is run over squared errors aggregated across all reading-time corpora, which mixes self-paced reading times, first-pass durations, and go-past durations with different units and noise levels; the test statistic, null hypothesis, and clustering are not specified. One or two large effects, such as Natural Stories SPR Shannon 29 vs. 72, could drive the reported p < 0.01 / p < 0.001 even if most per-corpus comparisons were null or reversed. The Discussion in §5 requires per-corpus or otherwise appropriately conditioned uncertainty quantification to support the caution it draws.","section":"§4.3 / Table 2"},{"comment":"The Introduction states that the MC method produces unbiased estimates, but this is not correct for the Rényi estimator in Eq. (5). That estimator applies a logarithm to a sample average of P(s)^{β−1}, so Jensen's inequality introduces finite-sample bias; for α = 1/2 the term P(s)^{−1/2} is particularly sensitive to rare sampled words and can produce high variance. The bootstrap coefficient-of-variation analysis in Appendix A.2 measures dispersion across resamples but does not measure bias or the influence of rare-word tail events. The Rényi results are an important part of Table 2, so the finite-sample properties of Eq. (5) should be acknowledged and, if possible, quantified.","section":"§3.3 / Eq. (5)"},{"comment":"The validity of the MC estimator as an approximation to true word entropy depends on the whitespace-partition sampling procedure producing samples from the LM's word-level conditional distribution. The paper follows Oh and Schuler (2024) by renormalizing over whitespace-initial tokens and defining the end-of-word probability as the sum over those tokens, but it does not validate on GPT2-small that this procedure yields the correct marginal word probabilities on the actual corpus tokenization. A small-scale exact enumeration or a comparison against the method of Pimentel and Meister (2024) would strengthen the claim that the MC estimates approximate true word entropy rather than another token-boundary quantity. The 20-token cap is acknowledged to underestimate true surprisal, but the magnitude of its effect on the entropy estimates is not quantified.","section":"Appendix A.1"}],"minor_comments":[{"comment":"The sentence 'These approximation are a lower bound' should read 'These approximations are a lower bound'.","section":"§3.3"},{"comment":"The variance analysis uses only the first story of the Natural Stories corpus; the authors should note that this corpus may not be representative of the lexical distributions in the eye-tracking corpora.","section":"Appendix A.2"},{"comment":"The regression setup would be easier to evaluate if the exact model formula and the method for obtaining held-out log-likelihoods (e.g., whether random effects were refit) were stated explicitly.","section":"§4.2 / Table 1"},{"comment":"The y-axis ranges of the two panels differ substantially; using a common scale or noting the difference in the caption would help readers compare the Shannon and Rényi coefficients of variation.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The skeptical reading of Table 2 is justified: the central empirical claim is not yet statistically established at the per-corpus level. However, the paper addresses a real methodological question and is probably fixable with additional uncertainty quantification and a more careful statement of what the aggregate permutation test can and cannot show. I would not reject on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if you work on entropy predictors for reading times, this paper is worth knowing. It applies Giulianelli et al.'s Monte Carlo sampling to next-word entropy, compares it against first-token entropy across eight corpus/measure settings, and finds the two predictors do not behave like interchangeable estimates. The point that first-token entropy is a lower bound that can distort entropy is not new (Pimentel et al. already said it), but the direct regression comparison is.\n\nCredit where due: the sampling procedure is described carefully, with the whitespace-initial token handling made explicit; code is released; the authors flag the limit of the 20-token cap and include a variance analysis; the POS figure nicely shows the distortion is larger for open-class words. Those are the right instincts.\n\nSoft spots, in proportion. The main one is statistical. Table 2 reports ΔLL point estimates with no standard errors or intervals. Several are close to zero or negative, so the sign of individual corpus comparisons is uninformative. The permutation test pools squared errors across corpora with different units (SPR times, first-pass, go-past), so a couple of large effects (Natural Stories) could drive the p-values. The claim that MC estimates are consistently better predictors is not supported by the Shannon rows: on 5 of 6 eye-tracking comparisons the first-token estimate does as well or better. It is the Rényi rows that more consistently favor MC. That conditional pattern is interesting but needs to be reported as such.\n\nOne technical point: the text says MC produces unbiased estimates, but the Rényi plug-in estimator in Eq. 5 is biased (consistent, but biased). The unbiasedness claim is correct for the Shannon estimator only. Easy fix: say 'consistent' or 'nearly unbiased for Shannon.'\n\nNone of this kills the paper. The core caution—do not assume first-token entropy is a safe proxy—is reasonable, and the descriptive divergence is enough to warrant a proper referee. This is for psycholinguistic methodologists and anyone using GPT-2-based entropy predictors. But the significance story is not yet established at the per-corpus level.\n\nRecommendation: send to peer review; ask for per-corpus confidence intervals or at least per-corpus significance tests, and a clearer separation of Shannon vs Rényi conclusions.","headline":"A useful empirical caution about first-token entropy approximations, but the headline divergence rests on shaky aggregate statistics.","tokens_in":8479,"tokens_out":3186,"would_cite":true,"duration_ms":40613,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"First-token entropy systematically underpredicts word entropy and yields reading-time predictions that diverge from Monte Carlo word-level estimates.","keywords":["contextual entropy","first-token approximation","Monte Carlo estimation","reading time","psycholinguistics","surprisal","Rényi entropy","subword tokenization"],"falsifier":"Compute exact word-level entropies for a set of contexts by exhaustively summing over all token sequences up to the length cap, or by using a word-level language model, and compare them to the MC estimates; if the MC estimates deviate systematically from the exact values, the sampler is biased and the paper's divergence claim collapses.","tokens_in":7446,"feed_emoji":"📖","tokens_out":7401,"duration_ms":73558,"temperature":0.7,"pith_summary":"Contextual entropy, the reader's uncertainty about the next word before seeing it, is usually estimated from a language model's probabilities over a word's first subword token because summing over all possible words is intractable. This paper argues that the first-token approximation is not just a slight underestimate: it systematically distorts which words look predictable, since multi-token words are common for open-class parts of speech. To test this, the authors build Monte Carlo estimates of word-level entropy that sample full words of variable token length, and compare both measures as predictors of reading times in self-paced reading and eye-tracking corpora. The two measures yield divergent regression results, with Monte Carlo word entropy fitting self-paced reading better and improving predictions in most eye-tracking evaluations. The paper concludes that first-token entropy may not be a reliable approximation for psycholinguistic modeling.","feed_headline":"Monte Carlo word entropy beats first-token entropy on reading times","feed_subtitle":"Sampling full multi-token words from GPT2 shifts which entropy predictor wins in reading-time models.","key_machinery":"The load-bearing mechanism is a Monte Carlo estimator of next-word entropy built from the language model's subword distribution. The vocabulary is split into whitespace-initial tokens and tokens without a leading whitespace; a word is sampled by drawing an initial token from the whitespace-initial set renormalized to sum to one, then drawing continuation tokens from the non-whitespace set together with an end-of-word symbol whose probability is the summed probability of all whitespace-initial tokens. The sampled word's surprisal is the sum of its token surprisals, and entropy is estimated by averaging over 512 samples; the same sampling supports a Rényi-entropy estimator. This procedure converts an intractable sum over an infinite word vocabulary into a tractable, unbiased expectation, and it is what lets the paper compare word-level entropy against first-token entropy.","core_discovery":"The paper's central claim is that first-token entropy is a lower bound on true word entropy and that the gap is large enough to change experimental conclusions. Using Monte Carlo sampling that draws whole words from GPT2-small by treating whitespace-initial tokens as word boundaries, the authors obtain unbiased word-level estimates of Shannon entropy and Rényi entropy with alpha = 1/2. In linear mixed-effects regressions over five reading-time corpora, replacing first-token entropy with Monte Carlo entropy changes the size and sometimes the sign of entropy's contribution, and permutation tests show the two predictors differ significantly. The authors read this as evidence that conclusions drawn from first-token entropy should be re-examined, and that word-level Monte Carlo estimates are a safer basis for studying anticipatory processing.","pith_inferences":["Beyond the paper, because distortion scales with multi-token word probability, larger subword vocabularies should shrink the first-token-versus-Monte-Carlo gap; this can be tested by rerunning the comparison with GPT2-large or different tokenizers.","Beyond the paper, the word-level Rényi advantage suggests the anticipatory strategy that best fits readers lies between Shannon averaging and max-probability tracking, which could be probed by fitting alpha as a free parameter.","Beyond the paper, the sampling estimator is not specific to entropy: the same whitespace-partition sampler could compute other word-level information measures, such as mutual information between a word and its context, without changing the data pipeline."],"forward_implications":["Self-paced reading results in this paper improve when entropy is measured at the word level rather than the first-token level, so word-level estimates should be preferred in studies of anticipatory processing.","First-token entropy not only underpredicts entropy but distorts it across parts of speech, so previous entropy effects estimated this way may mix tokenization artifacts with genuine processing difficulty.","Monte Carlo Rényi entropy with alpha = 1/2 outperforms first-token Rényi entropy on both self-paced and most eye-tracking evaluations, suggesting the word-level measure more consistently captures anticipatory processing.","Permutation tests over squared errors find the two predictors differ significantly, so the divergence is not just noise in one corpus."],"supporting_citations":[{"why":"Establishes that first-token entropy is a lower bound on word entropy, the baseline claim this paper challenges.","marker":"Pimentel et al. (2023)"},{"why":"Supplies the Monte Carlo sampling approach for continuation entropy and the claim that MC estimates are unbiased, which the paper adapts to next-word entropy.","marker":"Giulianelli et al. (2024)"},{"why":"Provides the whitespace-initial/non-whitespace vocabulary partition used to sample full words and to compute end-of-word probabilities.","marker":"Oh and Schuler (2024)"},{"why":"Shows how to compute the probability of a word from subword token probabilities, grounding the surprisal calculation for sampled words.","marker":"Pimentel and Meister (2024)"},{"why":"Provides GPT2-small, the language model whose conditional distributions supply all entropy and surprisal predictors.","marker":"Radford et al. (2019)"},{"why":"Supplies the linear mixed-effects regression machinery used to compare first-token and MC entropy as reading-time predictors.","marker":"Bates et al. (2015)"}],"fun_headline_variants":["First-token entropy undercuts reading-time analysis","Monte Carlo word entropy outperforms first-token proxy","Word-level entropy sampling flips predictor results","First-token entropy is a misleading proxy for word entropy","Unbiased word entropy changes reading-time conclusions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The MC comparison is only as good as the sampler: if sampling words by renormalizing whitespace-initial tokens does not reproduce the language model's true next-word distribution, the Monte Carlo estimates are biased and the divergence from first-token entropy would not reflect word-level truth.","fun_headline_variants_meta":{"raw":{"variants":["First-token entropy undercuts reading-time analysis","Monte Carlo word entropy outperforms first-token proxy","Word-level entropy sampling flips predictor results","First-token entropy is a misleading proxy for word entropy","Unbiased word entropy changes reading-time conclusions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000175,"raw_usage":{"total_tokens":1222,"prompt_tokens":818,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":331}},"tokens_in":434,"tokens_out":404,"duration_ms":5248,"temperature":1.0,"reasoning_tokens":331,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:56:39.275067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute exact word-level entropies for a set of contexts by exhaustively summing over all token sequences up to the length cap, or by using a word-level language model, and compare them to the MC estimates; if the MC estimates deviate systematically from the exact values, the sampler is biased and the paper's divergence claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides GPT2-small, the language model whose conditional distributions supply all entropy and surprisal predictors."}],"review_version":1}