{"id":"bf5c3298-9f60-4698-8dad-f13d9af7558c","arxiv_id":"2411.12728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLMs estimate about 20 bits of semantic information per clause in 20 spoken narratives, roughly half the total information per clause.","lead":"This paper uses large language models to measure how much meaning, not wording, each clause of a spoken story carries, reporting about 20 bits of semantic information per clause. It offers a practical way to quantify meaning-level information in natural communication, a quantity Shannon's theory left open.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under Eq. (5), IM,i must be nonnegative, yet the paper reports 14 negative clauses; this internal contradiction means the 20-bit/clause claim rests on an unvalidated prompt-based identification.","rationale":"The paper is a well-written proposal with a clear pipeline, released data, and several honest internal checks, and the human guessing experiment is a genuine partial validation: low-IM clauses tend to be predictable. But the central claim is a quantitative measurement of semantic information, and the defining equation itself is violated by the reported negative IM values. The reader's weakest assumption (paraphrase preserves meaning without leaking wording) is real, and the negative values are direct evidence of that violation. The SI's consistency tests (S.5–S.7) only establish that the estimator is reproducible across rephrasings and models, not that it measures P(Mi|c<i); all LLMs could share the same spurious conditioning on the paraphrase's wording. I therefore agree with the reader's CONDITIONAL verdict: the method is promising and potentially fixable, but the headline '20 bits of meaning per clause' is not yet established. A conditional acceptance with a request for the proposed paraphrase-fidelity and nonnegativity check is the right outcome; I would not reject the paper because the approach is constructive and the failure mode is identifiable and testable.","tokens_in":16402,"tokens_out":9763,"duration_ms":99859,"concrete_test":"Recompute IW,i for all clauses using a rephrased narrative whose clause-level meaning is independently verified by back-translation and human rating, with register and disfluencies matched to the original. If any IM,i remains negative, or if the mean IM shifts by more than ~3 bits from 20 bits, Prompt 2 does not implement Eq. (5) and the headline estimate is not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the LLM conditional probability computed under Prompt 2 equal P(Ci|Mi,c<i), the probability of the exact wording given only the clause meaning and prior context. This is not merely an untested idealization: under the paper's own decomposition, Eq. (5) gives P(Ci|c<i)=P(Mi|c<i)P(Ci|Mi,c<i). Since P(Mi|c<i)≤1, it follows that Ii = −log P(Ci|c<i) ≥ −log P(Ci|Mi,c<i) = IW,i, so IM,i = Ii − IW,i must be nonnegative for every clause. The paper reports 14 clauses with IM between −3 and 0 bits (SI §S.3). Those negative values are incompatible with Eq. (5) if Ii and IW,i were the conditional probabilities they are claimed to be; at least one of the two LLM estimates is not faithfully implementing the decomposition, and the paper provides no error analysis that would make 14 such violations harmless. The SI's own examples show why: the gpt-4 rephrasing of the Boy Scout narrative turns a swimming race into a running race and shifts register from colloquial to formal, so Prompt 2 conditions on a different meaning and on the paraphrase's specific lexical choices, not on Mi alone. Because Eq. (6) is the bridge from token log-probabilities to −log P(Mi|c<i), the headline 'about 20 bits of meaning per clause' is not supported by the current estimator. The consistency checks in SI §S.5–S.7 show reproducibility across rephrasings and models, but they do not test whether the estimated quantity is the semantic information defined in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an information-theoretic decomposition of clause-level information in narratives into semantic information and wording information, and uses LLM conditional probabilities to estimate both terms. The semantic information of each clause is computed as the difference between the total information of the clause and its wording information, where the latter is estimated by conditioning on a ChatGPT-generated rephrasing of the narrative that is intended to fix the clause's meaning. Applied to 20 Labov personal-experience narratives, the method yields an average semantic information of about 20 bits per clause, roughly half of the total information per clause. The paper reports that clauses humans can predict from context indeed have low semantic information, and that the initial clauses of a narrative carry more information unless the interviewer's guiding question is provided.","tokens_in":16745,"tokens_out":8720,"duration_ms":77234,"significance":"If the proposed identification is valid, the paper offers a novel operational measure of meaning-level information in natural language, bridging Shannon information and semantics, with potential applications to dialogue and summarization. The authors provide open data, a human guessing benchmark, and cross-model consistency checks (SI S.6-S.7), which are valuable assets. However, the central equality between the LLM probability under the rephrasing prompt and P(C_i|M_i,c_<i) is assumed rather than derived, and the paper's own examples and numerical results (negative semantic information, three human-predictable clauses with 10-12 bits) show that the estimator is not faithful to this equality. The headline average of ~20 bits/clause therefore rests on an unvalidated identification.","major_comments":[{"comment":"The decomposition in Eq. (5) implies that I_M,i = I_i - I_W,i >= 0 for every clause, because I_i = -log P(C_i|c_<i) = -log P(M_i|c_<i) + I_W,i and P(M_i|c_<i) <= 1. Yet SI S.3 reports 14 clauses with negative semantic information, between -3 and 0 bits. This is an internal contradiction if the LLM estimates faithfully implement the conditional probabilities in Eq. (5). The paper describes these as 'wrong estimates' but does not provide an error model or a corrected estimator, and the headline average of about 20 bits/clause is computed over all clauses including these negative values. The authors should either enforce non-negativity with a constrained estimator or provide a quantitative bias analysis showing that the negative values are within the estimation noise and do not affect the reported average.","section":"Section 2, Eqs. (5)-(6); SI S.3"},{"comment":"The central identification P(C_i|M_i,c_<i) = P(C_i | prompt containing a ChatGPT rephrasing) is asserted rather than derived or validated. The paper's own Boy Scout example (SI S.2.2) shows that the rephrasing changed the meaning (a swimming race became a running race) and shifted the register from colloquial to formal. Therefore Prompt 2 conditions on a different meaning and on style features, violating the 'single meaning approximation' and the assumption that the rephrasing leaks nothing about phrasing. The consistency checks in SI S.6 demonstrate reproducibility across rephrasings, but reproducibility does not establish correctness; a direct validation of the equality, for instance using human-verified meaning-preserving paraphrases or measuring whether I_W estimates are invariant to style variations, is needed before the reported 20 bits/clause can be accepted.","section":"Section 2, SI S.2.2-S.2.3"},{"comment":"Three clauses that were correctly predicted by human participants have estimated semantic information between 10 and 12 bits. The paper acknowledges that this 'indicates that our method sometimes results in wrong estimates of information,' but offers no explanation or remedy. Because the method is explicitly intended to rank clause predictability (clauses with low I_M should be predictable), these outliers are a direct counterexample for a non-negligible fraction (3/23) of the human-predictable clauses. The authors should report the full distribution of I_M for the 23 human-predictable clauses, including the fraction exceeding a threshold, and discuss whether the method can still be used for ranking despite these failures.","section":"Section 3, Fig. 3A"},{"comment":"The headline estimates use the full rephrasing in Prompt 2, which includes rephrased clauses after C_i, so the model's probability is conditioned on information about the future of the narrative, whereas the decomposition in Eq. (5) conditions only on c_<i. The partial-rephrasing test in SI S.5 reports that the wording information differs by -1.1 ± 2.3 bits between the partial and complete rephrasing conditions; this non-zero difference shows a systematic effect of future context, and the sign of the difference appears inconsistent with the stated interpretation that future context further constrains the meaning. The paper neither corrects the headline numbers for this bias nor justifies why the full-rephrasing estimate is preferable for a quantity defined only with respect to preceding context.","section":"Section 2 and SI S.5"}],"minor_comments":[{"comment":"The paragraph beginning 'In Fig. 1A, we show the cumulative sums of I_i...' appears twice in the main text, once before and once after Eq. (6); the duplication should be removed.","section":"Section 2"},{"comment":"In Fig. 1B, the inset histogram's axis label is truncated ('Sem. Info. [bits]'); please provide a complete label, and ensure all subplot axes are clearly labeled.","section":"Figure 1"},{"comment":"Example 3 (Gloria Stein) is difficult to read because the original and rephrased clauses are not aligned in a table; consider formatting it as a two-column table for clarity.","section":"SI S.2.2"},{"comment":"The token-level information I_x,i is defined but the relationship to the clause-level information I_i is not stated explicitly; please add a sentence clarifying that I_i is the sum of I_x,i over the tokens of the clause.","section":"SI S.2.1"},{"comment":"Reference [18] is to a preprint; if a published version exists, it should be cited instead or in addition.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The authors are transparent about the method's limitations, but the negative semantic information and the three high-information predictable clauses are not minor caveats; they directly undermine the central claim. In my view, the manuscript requires substantial additional validation before the headline 20 bits/clause can be considered reliable. I would also encourage the authors to remove the password protection from the data zip to facilitate reproduction, and to consider whether the title and abstract overstate the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely new operational idea—estimate the wording information of a clause by conditioning an LLM on a rephrased narrative, then define semantic information as total minus wording. It is honestly written, ships data and code, and the authors run several sanity checks. But the central quantitative claim is not supported by the estimator as it stands. I read the stress-test note and think it is correct: under Eq. (5), I_M,i is necessarily nonnegative, yet the paper's own SI reports 14 clauses with I_M between -3 and 0 bits. Those negative values can only mean the two LLM probabilities are not faithful to the decomposition. The paper waves at this in passing but offers no error model or convergence check that would make 14 violations harmless. The same estimator puts three human-predictable clauses at 10-12 bits, which is another sign the number is off.\n\nWhat is genuinely new: the equivalent decomposition (Eq. 5) is already in the cited literature, but the prompt-based estimator and the first measurement of semantic bits per clause on natural narratives are new. The robustness checks are real: partial vs. full rephrasing, second rephrasing, several LLMs, and a human guessing experiment. That is more than most papers of this type do, and it earns the work a careful reading.\n\nThe soft spots are not minor. The bridge from token log-probabilities to -log P(M_i|c_<i) depends on the paraphrase capturing exactly the meaning and nothing else. The paper's own Boy Scout example shows meaning drift (swimming race becomes running race), and the style shift from colloquial to formal is acknowledged. The headline average of about 20 bits/clause also has no error bars. So the conclusion that roughly half the information in a clause is meaning is not established.\n\nThe paper is for people working on semantic information theory, LLM evaluation, and psycholinguistics. It deserves a serious referee because the idea is worth testing and the authors did a lot of legwork. A referee should demand a corrected estimator that is validated on cases where the true semantic information is known, and the negative-values problem must be resolved, not simply acknowledged.\n\nI would send it to review, but the review should be frank.","headline":"A promising but unvalidated estimator for semantic information in narrative: the headline 20 bits/clause is not supported by the paper's own numbers, but the idea and the data are worth a serious referee.","tokens_in":17250,"tokens_out":2949,"would_cite":false,"duration_ms":28795,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["94A17","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the semantic information of a clause can be estimated as the difference between total and wording information computed with LLM probabilities, yielding about 20 bits per clause on average.","keywords":["semantic information","information theory","large language models","narrative","clause","entropy","meaning","Labov narratives"],"falsifier":"Construct a narrative and two rephrasings: one that preserves all meanings and one that swaps a single concrete detail, such as changing a swimming race to a running race in the Boy Scout story. If the semantic-information estimate for the altered clause does not decrease substantially relative to the faithful rephrasing, the meaning-fixing assumption fails in a directly observable way. A companion check: among clauses the LLM deems predictable, human guessers should mostly agree; the paper already reports 8 clauses ChatGPT predicts but no human participant could, which is a concrete place to look.","tokens_in":16159,"feed_emoji":"🧠","tokens_out":5155,"duration_ms":46796,"temperature":0.7,"pith_summary":"The paper tries to establish that the information carried by the meaning of a clause in natural narrative can be quantified in bits, and that large language models can supply the needed probabilities. It defines the semantic information of clause $C_i$ as $I_{M,i} = I_i - I_{W,i}$, where $I_i$ is the total information of the clause under the LLM and $I_{W,i}$ is the wording information estimated after a rephrased version of the story is presented as fixing the meaning. Applied to 20 Labov narratives, the measure yields about 20 bits of semantic information per clause, roughly half of the about 40 bits per clause of total information. If this is right, the paper provides an operational, LLM-based measure of meaning-level information in ordinary spoken stories.","feed_headline":"Semantic information in stories: about 20 bits per clause","feed_subtitle":"Paper separates meaning from wording in 20 spoken narratives, matching Shannon-style entropy estimates.","key_machinery":"The load-bearing decomposition is $P(C_i|c_{<i}) = P(M_i|c_{<i}) P(C_i|M_i, c_{<i})$, which rewrites the probability of a clause's wording as the probability of its meaning times the probability of that wording given the meaning. Under the single-meaning approximation this yields $I_{M,i} = I_i - I_{W,i}$. The wording information $I_{W,i}$ is obtained by asking the LLM for conditional probabilities of the original clause tokens in a prompt that presents the rephrased narrative followed by the original and states that they tell the same narrative; the rephrasing is produced by GPT-4 with a clause-preserving prompt.","core_discovery":"The central claim is that, under a 'single meaning approximation' in which each clause has one meaning given its context, the probability of a clause's wording factors into the probability of its meaning times the probability of that wording given the meaning. This yields the identity $I_{M,i} = I_i - I_{W,i}$, so semantic information is the total information of the clause minus the information that is due to the particular wording once the meaning is fixed. The paper estimates both terms with Llama-3.1-70B token probabilities, using a GPT-4-generated rephrasing of the narrative to fix the meaning in the wording-information prompt. The result for 20 Labov narratives is an average semantic information rate near 20 bits per clause, with clauses ranging from roughly 0 to 100 bits and 14 clauses showing negative values that the paper attributes to limitations of the meaning-fixing procedure.","pith_inferences":["My inference: the same measure could be applied to dialogues, where shared context should lower semantic information per turn; a drop in $I_M$ across conversational turns would support the paper's claim that context constrains meaning.","My inference: if the method is sound, it provides a quantitative handle on 'gist' memory, consistent with the well-known finding that people retain meaning while forgetting exact wording; one could test this by comparing $I_M$ of recalled versus forgotten clauses in memory experiments.","My inference: the 14 negative-$I_M$ clauses are a diagnostic resource; analyzing them could reveal exactly which paraphrase failures (lexical, syntactic, or stylistic) push the estimate below zero and suggest improved prompts."],"forward_implications":["The measured rate of about 20 semantic bits per clause, with about 40 total bits per clause, means roughly half of the information in each clause is carried by wording and half by meaning.","Clauses whose next meaning can be guessed from preceding context mostly have semantic information near zero, so the measure tracks human predictability.","On average the first clause of a narrative is the most informative when the opening question is not supplied; supplying the interviewer's guiding question removes this trend.","The estimates are similar when computed with partial rephrasings, with a second independent rephrasing, or with different LLMs (Llama-3.1-8B, Llama-3.1-8B-Instruct, Mistral-Nemo-12B)."],"supporting_citations":[{"why":"provides the 20 personal-experience narratives that are the dataset for all measurements.","marker":"[1]"},{"why":"supplies the clause segmentation into independent and dependent clauses used as the unit of analysis.","marker":"[2]"},{"why":"provides the Shannon entropy-of-English estimate (about 1 bit per character) that the paper compares with its clause-level rates.","marker":"[3]"},{"why":"Llama-3.1-70B-Instruct is the LLM whose token probabilities give both total and wording information.","marker":"[14]"},{"why":"GPT-4 is used to generate the rephrased narratives that are supposed to fix clause meanings.","marker":"[17]"},{"why":"introduces the semantic uncertainty decomposition over phrasings that the paper adapts into its single-meaning probability sum.","marker":"[15]"}],"fun_headline_variants":["Meaning measured: ~20 bits per clause in stories","LLMs quantify meaning: stories carry ~20 bits per clause","Clause-level meaning: ~20 bits, via LLM rephrasing","How many bits is meaning? About 20 per clause in narratives","From wording to meaning: LLMs estimate ~20 bits a clause"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the rephrased narrative placed in the second prompt makes the LLM's conditional probability of the original wording equal to $P(C_i|M_i, c_{<i})$; that is, the paraphrase fixes exactly the meaning of each clause and reveals nothing about its phrasing, style, or register, and the paper's own examples show meaning drift and 14 clauses with negative semantic information indicate the assumption is violated in practice.","fun_headline_variants_meta":{"raw":{"variants":["Meaning measured: ~20 bits per clause in stories","LLMs quantify meaning: stories carry ~20 bits per clause","Clause-level meaning: ~20 bits, via LLM rephrasing","How many bits is meaning? About 20 per clause in narratives","From wording to meaning: LLMs estimate ~20 bits a clause"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2393,"prompt_tokens":822,"completion_tokens":1571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":1481}},"tokens_in":438,"tokens_out":1571,"duration_ms":11901,"temperature":1.0,"reasoning_tokens":1481,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:12:18.575576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a narrative and two rephrasings: one that preserves all meanings and one that swaps a single concrete detail, such as changing a swimming race to a running race in the Boy Scout story. If the semantic-information estimate for the altered clause does not decrease substantially relative to the faithful rephrasing, the meaning-fixing assumption fails in a directly observable way. A companion check: among clauses the LLM deems predictable, human guessers should mostly agree; the paper already reports 8 clauses ChatGPT predicts but no human participant could, which is a concrete place to look.","supporting_citations":[],"review_version":1}