{"id":"cbc145da-e1d9-41b1-b6fd-616f587c3bc5","arxiv_id":"2607.17447","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A prespecified semantic map plus held-out calibration can turn LLM token probabilities into calibrated posterior estimates over declared states, with bounded error and valid coverage in tested settings.","lead":"This paper defines a semantic map: a prespecified statistical bridge from a language model's word probabilities to probabilities over meaningful states such as diagnoses or market regimes. It tests that bridge on market text and controlled simulations, finding that calibrated language probabilities beat the model's printed numerical confidence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero empirical pairwise lower modulus and O(7) affine residuals mean Theorem 3's key observability condition is not certified; recovery claim rests on finite-design averages.","rationale":"The paper is careful, prespecified, and unusually honest about limitations. The theorems are correctly stated; the issue is the gap between the theorem's condition and the empirical certification. The reader reached CONDITIONAL, and my analysis supports that verdict. I do not see an internal mathematical error in the proofs. The main risk is that the headline 'recovers held-out posteriors' overstates the strength of the evidence because the key modulus is not merely uncertified but reported as zero in the empirical pairwise diagnostic. However, the paper itself flags this and does not present a uniform identification claim, so this is a reason for conditional acceptance, not rejection. The code/data unavailability is related but secondary; the zero-modulus issue is more directly load-bearing.","tokens_in":29082,"tokens_out":8553,"duration_ms":97304,"concrete_test":"Using the archived calibration observations for both models, compute the empirical pairwise lower modulus from scenario-mean lexical log ratios: min over i≠j of ||λ̄_i−λ̄_j|| / ||ℓ_i−ℓ_j||. Identify the scenario pairs attaining the minimum. Then construct a held-out challenge set by generating new reference-model evidence whose posterior log-ratios interpolate between those pairs, run the same frozen semantic map and calibrated inverse, and measure recovery error along that direction. If error grows systematically as the interpolated direction approaches the null pair and the conformal sets fail to cover, the finite-design observability claim is falsified; if error remains small and coverage holds, the zero modulus is an artifact of noise on individual pairs and the residual concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recovery guarantee is Theorem 3 (§2.2.3), whose error bound is inverse in the lower modulus c_u. The paper's own observability interpretation (§3.2) reports that the empirical pairwise lower modulus is 'effectively zero' for GPT-4.1-mini and 'zero at machine precision' for GPT-4o-mini, with affine residual RMSEs of 7.62 and 7.35 in lexical log-ratio units. If the pairwise modulus over the sampled calibration design is zero, then some pairs of reference posteriors produce indistinguishable lexical measurements; no inverse can recover both, and Theorem 3's bound is vacuous. The positive bootstrap lower bounds on the smallest singular value (1.686, 1.600) are properties of the fitted affine matrix, not of the true forward regression h_u. Large residuals mean the affine map is a poor approximation, so the fitted map's conditioning does not transfer to the actual inverse problem. The empirical claim of posterior recovery is therefore supported only by average held-out error and split-conformal coverage, not by the theorem the paper advertises. This is acknowledged in the limitations, but the abstract's phrasing ('recover held-out posteriors ... auditable posterior estimate') does not carry the qualification that the central identifiability condition is neither certified nor, by the paper's own diagnostics, satisfied on the sampled design.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a 'semantic map': a prespecified statistical bridge from an LLM's observable continuation probabilities over verbal responses to a reference posterior over a finite set of declared states. The construction separates the reference experiment from the language experiment, defines a lexical composition p_u through semantic coarsening φ_u and an unexpressed-mass category ⊥, and proposes held-out calibration of an inverse map ψ_u to estimate the reference posterior. The theoretical part derives an approximation decomposition (Theorem 1), a presentation-stability bound (Theorem 2), an inverse-recovery bound based on a lower-modulus observability condition (Theorem 3), and a sequential-filtering error bound (Theorem 4). The empirical part compares continuation-derived lexical probabilities with numerically elicited probabilities on market text, and evaluates posterior recovery and conformal coverage in a controlled three-state experiment with two fitted LLMs. The paper is unusually careful about prespecification, scenario clustering, and limitations, but the central empirical certification is confined to a constrained enum response grammar and does not certify the key observability condition of Theorem 3.","tokens_in":29435,"tokens_out":4497,"duration_ms":49656,"significance":"If the central claim held as stated, the paper would be a substantial contribution: it provides a declared estimand, a decomposition of semantic, probability-mass, and inverse-recovery error, and a held-out validation protocol for converting LLM token probabilities into auditable state posteriors. The strengths are genuine: the semantic coarsening is prespecified, calibration and testing use disjoint scenarios, uncertainty is clustered by scenario, the experiments are replicated across two fitted models, and the limitation statements are unusually explicit. However, the significance is currently limited by a gap between the advertised theorem-driven recovery guarantee and the empirical evidence, which establishes only finite-design, average-case recovery under a constrained grammar. The contribution is therefore promising but needs reframing before the abstract-level claims are supported.","major_comments":[{"comment":"The central recovery guarantee of Theorem 3 is conditional on a positive lower modulus c_u. The paper's own diagnostic reports that the empirical pairwise lower modulus is 'effectively zero' for GPT-4.1-mini and 'zero at machine precision' for GPT-4o-mini, with affine residual RMSEs of 7.62 and 7.35 in lexical log-ratio units. If the pairwise modulus on the sampled design is zero, there exist reference posteriors that are indistinguishable from the mean lexical measurement, and Theorem 3's bound 2δ/c_u is vacuous. The positive bootstrap lower bounds on the smallest singular value (1.686 and 1.600) are properties of the fitted affine matrix, not of the true forward regression h_u, and the large residuals show the affine approximation is poor. Consequently, the abstract's claim that the method 'recover[s] held-out posteriors with valid uncertainty coverage' as an 'auditable posterior estim","section":"§3.2, 'Observability interpretation'; §2.2.3, Theorem 3"},{"comment":"The general theorems are stated for the unrestricted free-text law Q_u, but the experiments evaluate only the constrained enum grammar Q^G_u with three enum codes plus an explicit residual, under a completeness threshold of 0.999999. The paper acknowledges this in Section 2.1.1 and Section 3.2.1, but the abstract and several conclusions state without qualification that 'language-derived probabilities outperform printed numerical probabilities, recover held-out posteriors with valid uncertainty coverage.' This overstates the scope of the empirical certification. The abstract and conclusion should explicitly state that the recovery results are for the constrained response grammar used in the controlled experiment and do not extend to unrestricted free-text continuations.","section":"§2.1.1 (constrained law Q^G_u); §3.2; Abstract"}],"minor_comments":[{"comment":"The theorem assumes existence of the argmin b̂ℓ but does not state a compactness or attainment condition. The appendix proof mentions compactness of L; this assumption should be stated in the theorem itself.","section":"§2.2.3, Theorem 3"},{"comment":"The split-conformal coverage values 0.94 and 0.90 are reported on 100 test scenarios without a confidence interval. With 100 scenarios, the binomial standard error is about 0.024–0.030, so nominal coverage is plausible but not tightly certified; this should be noted or an interval provided.","section":"Table 7"},{"comment":"The raw entropy correlation r ≈ −0.001 and calibrated posterior entropy r = 0.742 are reported without confidence intervals. Given that these are descriptive and based on a modest number of scenarios, an interval or at least a statement of uncertainty would avoid overinterpretation.","section":"§3.2, entropy association"},{"comment":"The distinction between Q_u and Q^G_u is central, but the notation is introduced only in Section 2.1.1 and then not always reused. For readability, consider using Q^G_u consistently in the experimental sections and in Table 4 to remind readers which law is being certified.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically careful and the held-out design is a real strength. The main risk is that the abstract and conclusion claim theorem-backed recovery while the paper's own diagnostics show the key observability modulus is zero on the sampled design. This is fixable by reframing the central claim as finite-design, average-case recovery under the constrained grammar, but the revision is substantive because it changes what the paper advertises as its contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about arXiv:2607.17447. First, it is a serious and unusually honest attempt to turn LLM continuation probabilities into calibrated posteriors over declared states: the semantic-map framing, the error decomposition, and the held-out calibration protocol are genuinely new as a package. Second, the gap between the abstract and what is actually certified is real, though narrower than a stress-test might suggest — the paper itself is candid about most of it.\n\nWhat is new: instead of treating semantic grouping as an ad hoc step, the author embeds it in a semiparametric inverse measurement model. The pushforward construction, the finite-lexical-approximation bound (Theorem 1), the perturbation bound, and the filtering bound are standard tools, but assembled coherently. The experimental design is a model of prespecification: scenario clustering, frozen partitions, explicit rejection criteria, and a controlled experiment with exact reference posteriors. The cross-model replication on two GPT models is good practice. The four-component decomposition — semantic mismatch, probability error, unexpressed mass, inverse recovery — is a useful contribution by itself.\n\nSoft spots, in order of importance. The empirical certification is for a constrained enum grammar Q^G_u with three codes plus a residual, not the unrestricted free-text law that the framework defines. The paper states this explicitly, so it is an announced scope limitation rather than a hidden one; but the abstract's phrasing could mislead a casual reader.\n\nThe deeper issue is Theorem 3. The recovery bound needs a positive lower modulus c_u, and the paper's own observability section reports the empirical pairwise lower modulus as effectively zero for GPT-4.1-mini and zero at machine precision for GPT-4o-mini, with affine residual RMSEs around 7.6 in lexical log-ratio units. So the theoretical inverse guarantee is not certified on the sampled design; the positive singular-value bootstrap bounds describe the fitted affine map, not the true forward regression. The paper acknowledges this in the limitations and in Section 2.2.3. What the experiments actually demonstrate is average held-out recovery and split-conformal coverage on a finite design, not uniform identifiability. That is still a useful result, but it should be framed as an empirical template plus conditional theory, not a theorem-backed recovery guarantee.\n\nMinor: no data or code artifacts are currently available, only promises. The target posterior is defined by the authors' reference model, which is fine for a template but limits external validation.\n\nVerdict: worth a serious referee. A careful referee should push for a sharper statement of which conditions are empirically verified and for release of the artifacts. The paper deserves to enter the literature as a methodological contribution, with the rhetoric tightened to match the evidence.","headline":"A serious, unusually honest attempt to turn LLM continuation probabilities into calibrated posteriors over declared states, but the theoretical recovery guarantee is not certified by the experiments — the empirical claims rest on a constrained grammar and finite-design diagnostics.","tokens_in":29860,"tokens_out":3016,"would_cite":true,"duration_ms":29073,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F15","62G05","62H12"],"pacs":[],"model":"deepseek-v4-flash","headline":"A language model's probabilities over phrases can be turned into calibrated, auditable posterior estimates over meaningful states when a semantic map and held-out calibration are fixed in advance.","keywords":["semantic uncertainty","language-model probabilities","posterior calibration","inverse problems","compositional data","semiparametric inference","conformal prediction","semantic grouping"],"falsifier":"Find a pair of evidence values inside the declared operating domain whose exact reference posteriors differ materially but whose lexical compositions under the same prompt and semantic map are (nearly) identical, so the fitted affine map's smallest singular value is effectively zero on that pair. Alternatively, apply the published semantic map and calibration to a different language-model checkpoint or to unrestricted free-text responses and check whether held-out conformal coverage falls below the nominal level or held-out error exceeds the reported bounds.","tokens_in":28959,"feed_emoji":"🎯","tokens_out":5465,"duration_ms":49356,"temperature":0.7,"pith_summary":"The paper tries to establish that a language model's probabilities over complete verbal continuations—rather than the numbers it prints when asked for confidence—can be turned into a calibrated posterior over meaningful states, if one fixes in advance how phrases map to states and validates the map against a reference posterior. The method is a 'semantic map': a prespecified grouping of phrase probabilities into a lexical composition, followed by an inverse calibration fitted on held-out scenarios and expressed in additive log-ratio coordinates. The paper derives error bounds, identifiability conditions, presentation-stability bounds, and a sequential-filtering bound for this recovery. Empirically, language-derived probabilities outperform numerical elicitation, recover exact held-out posteriors with nominal conformal coverage, are stable under paraphrase, and move in the right direction when evidence changes. What is at stake is replacing unverified confidence statements with an auditable statistical measurement.","feed_headline":"Calibrated phrase odds recover held-out posteriors","feed_subtitle":"A prespecified semantic map plus held-out calibration yields valid uncertainty sets in tests on two fitted language models.","key_machinery":"The carrying object is the semantic map, defined as a prespecified measurable coarsening φ_u of the continuation space into K declared semantic states plus an unexpressed symbol ⊥, followed by a calibrated inverse ψ_u fitted in additive log-ratio (alr) coordinates. The estimator is the composition bπ_u = alr^{-1}∘ψ_u∘alr∘p_u∘C_u, where p_u is the normalized pushforward of the language law under φ_u. The map's load-bearing property is the lower modulus c_u: the minimal separation the language channel preserves between different reference posterior log-ratios. It converts a bounded observation error δ into a bounded recovery error 2δ/c_u, and its reciprocal controls how strongly inversion ampl","core_discovery":"The central claim is that the observable, prompt-dependent distribution over verbal responses can serve as a measurement of a reference posterior over a finite set of declared states. The construction fixes a semantic coarsening φ_u (complete continuations mapped to states or to an unexpressed category), forms the normalized lexical composition p_u = pushforward of Q_u under φ_u, and calibrates an inverse map ψ_u in alr coordinates to obtain bπ_u = alr^{-1}∘ψ_u∘alr∘p_u∘C_u. Under a positive lower-modulus condition the recovery error is bounded by 2δ/c_u (or δ/κ_u in the affine case), so observability of the language channel is exactly what protects against amplified noise. The paper's experi","pith_inferences":["The constrained three-code response grammar used in the controlled experiment may be much easier to calibrate than unrestricted free text; using the method on open-ended prose will likely require richer semantic partitions and explicit handling of unexpressed mass, and the reported results do not certify that setting.","Because the forward map is certified only as affine on the sampled design, posterior estimates near the simplex boundary or outside the calibrated region could carry larger inversion error than the average held-out metrics suggest.","The dependence on a frozen fitted language distribution implies that model updates or prompt-service changes invalidate the calibration; deployed systems would need ongoing recalibration rather than a one-time semantic map.","The entropy contrast (raw word entropy nearly uncorrelated with posterior entropy, calibrated entropy correlated around 0.74) suggests that using raw token entropy as an uncertainty measure is unsupported; the semantic map plus calibration is the effective bridge."],"forward_implications":["Language-model continuation probabilities can be treated as a measurement channel with a declared estimand, making posterior estimates auditable rather than ad hoc.","Uncertainty sets with nominal coverage can be attached to language-derived state probabilities when calibration and conformal radii are computed on held-out scenarios.","Information-preserving rewording can be validated as a nuisance factor, while reordering that preserves the facts is shown to be consequential.","Sequential Bayesian updating becomes defensible when the reference filter is contractive and one-step recursion defects are observable.","The same declared-map-plus-calibration template extends to auditing classifications and recommendations beyond probability reporting."],"fun_headline_variants":["Phrase odds beat printed confidences for real-world posteriors","Recover hidden states via reworded phrase probabilities","Language odds calibrated to reference posteriors win","Semantic map turns phrase odds into auditable uncertainty","Phrase-level uncertainties match held-out posterior coverage"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the forward map from reference posterior log-odds to lexical log-odds is identifiable and stable—has a positive lower modulus and is well approximated by the fitted affine class—on the intended operating domain, and the experiments certify this only on a sampled design for a constrained three-code response grammar rather than for unrestricted free text.","fun_headline_variants_meta":{"raw":{"variants":["Phrase odds beat printed confidences for real-world posteriors","Recover hidden states via reworded phrase probabilities","Language odds calibrated to reference posteriors win","Semantic map turns phrase odds into auditable uncertainty","Phrase-level uncertainties match held-out posterior coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":2963,"prompt_tokens":794,"completion_tokens":2169,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":2108}},"tokens_in":538,"tokens_out":2169,"duration_ms":15967,"temperature":1.0,"reasoning_tokens":2108,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:55:26.908649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a pair of evidence values inside the declared operating domain whose exact reference posteriors differ materially but whose lexical compositions under the same prompt and semantic map are (nearly) identical, so the fitted affine map's smallest singular value is effectively zero on that pair. Alternatively, apply the published semantic map and calibration to a different language-model checkpoint or to unrestricted free-text responses and check whether held-out conformal coverage falls below the nominal level or held-out error exceeds the reported bounds.","supporting_citations":[],"review_version":1}