{"id":"6206c823-241c-4e11-b355-12ba935ad580","arxiv_id":"2505.17135","paper_version":4,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Using a log-linear model and Jacobian analysis, the paper claims isotropy in LLM hidden embeddings stabilizes the softmax partition function and improves time-series forecasting, with illustrative experiments on five models and 22 datasets.","lead":"This paper asks whether the geometric spread of large language model embeddings, called isotropy, explains how well the models can forecast numerical time series. It argues that more isotropic embeddings stabilize the models' internal scoring and improve numerical performance, but the supporting theory and evidence are weaker than the claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 5 isotropy is invariant to the global logit shift that Theorem 1 exploits, so it cannot resolve softmax shift-invariance.","rationale":"The paper's abstract and conclusion claim that isotropy in the contextual embedding space resolves the shift-invariance of softmax and provides a performance guarantee for numerical downstream tasks. The reader's verdict targets the missing bridge between the theoretical isotropy of Eq. 5 and the empirical cosine-similarity isotropy of Eq. 6. That gap is real, but there is a more basic problem: even Eq. 5, taken on its own terms, does not resolve shift-invariance. Because I is defined as a ratio of partition-function values, a global additive shift of all logits multiplies every partition function by the same constant and leaves I unchanged. Theorem 1's failure construction is precisely such a global shift. Hence a model with perfectly isotropic representations in the paper's own sense can still suffer the exact downstream failure the paper says isotropy prevents. This is not a matter of missing empirical validation; it is an internal inconsistency in the central argument. One possible defense is that the shifted logits in Theorem 1 might not be realizable by the inner-product student model in Eq. 3, since a constant offset must lie in the column space of the token-embedding matrix. But the proof of Theorem 1 does not verify this representability, so the theorem's existence claim is either unproved or, if the shift is representable, its impact on Eq. 5 is invisible. Either way, the performance-guarantee claim is unsupported. The empirical correlation between cosine-similarity isotropy and NMSE may still be a useful heuristic, and a revised paper that removed the guarantee language, fixed or replaced Theorem 1, and connected Eq. 6 to a well-defined notion of partition-function stability would be worth revisiting. As it stands, the central claim fails under direct scrutiny, so the REJECT verdict is appropriate. My concern is close to the reader's but more fundamental, so I mark partial agreement rather than full agreement on the weakest assumption.","tokens_in":16752,"tokens_out":6808,"duration_ms":62498,"concrete_test":"Analytically verify the shift-invariance of Eq. 5: for any finite set of contexts with logits z_i(k,l), compute I({z}) and I({z+τ}) for a constant τ such as τ = −10. If the values coincide, the isotropy ratio cannot distinguish the ground-truth model from the shifted student model of Theorem 1. As a direct instantiation, take a ground-truth model in which all Z(k,l) are equal, so I=1, apply the Appendix A choice of τ below min_j b_j − max_j z_i(k,l), and check that f̂=0 while I remains 1. This settles whether Eq. 5 can provide the claimed resolution of softmax shift-invariance.","verdict_should_be":"REJECT","load_bearing_attack":"The central guarantee rests on Eq. 5, where isotropy is defined as I({ψ(k)}) = min_C Z(k,l) / max_C Z(k,l), with I≈1 said to make the partition function stable and thereby resolve softmax shift-invariance. This definition is scale-invariant: replacing every logit z_i by z_i + τ multiplies every partition function by e^τ, so the min/max ratio and I are unchanged. The construction in Theorem 1 (Appendix A) is exactly such a global shift: ẑ_i = z_i + τ with τ chosen so that σ(ẑ_i − b_i)=0 for every i. Therefore a student model built from that construction has the same I as the ground-truth model, including I≈1 when the ground-truth model is isotropic, while its downstream prediction f̂(k,l)=0. Isotropy as defined is thus compatible with the very failure the paper claims to prevent; the claimed performance guarantee does not follow. The empirical metric in Eq. 6 has the same blind spot: mean-subtracted cosine similarity is unchanged by translating an entire cluster, so if the global logit shift is realized by translating context embeddings, the measured isotropy also fails to detect the failure. The paper therefore provides no formal link from either Eq. 5 or Eq. 6 to the absence of harmful logit shifts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to explain when the next-word prediction capability of LLMs can be adapted to numerical forecasting tasks. It models next-token probabilities with a log-linear softmax model, defines a numerical downstream task as a ReLU-thresholded linear function of logits, proves that without structural constraints a student model can match the pretraining distribution while making downstream prediction zero (Theorem 1), and proposes isotropy of contextual embeddings as the missing structural constraint. The paper defines isotropy via a min/max partition-function ratio (Eq. 5), derives spectral properties of self-attention (Lemma 1, Theorem 2), and reports experiments measuring cluster-level mean-subtracted cosine similarity (Eq. 6) against NMSE across 12 real and 10 synthetic datasets.","tokens_in":17005,"tokens_out":5556,"duration_ms":46274,"significance":"If the results were correct, the paper would give a practically useful diagnostic: a cosine-similarity-based isotropy score that indicates when an LLM's embeddings are safe for numerical downstream tasks. The paper is also honest about limitations and provides a detailed experimental protocol. However, the central theoretical claim is not established: the definition of isotropy in Eq. (5) is invariant to the exact logit shift used in Theorem 1, and the experimental metric in Eq. (6) is never formally connected to Eq. (5) or to downstream error. The empirical observations, while suggestive, are presented example-by-example without aggregate statistics, so they do not by themselves support a performance guarantee.","major_comments":[{"comment":"The definition I({ψ(k)}) = min_C Z(k,l) / max_C Z(k,l) is invariant under the global logit shift used in Theorem 1: replacing every logit z_i(k,l) by z_i(k,l)+τ multiplies every partition function by e^τ and leaves the min/max ratio unchanged. Therefore a student model constructed as in Appendix A has exactly the same isotropy value as the ground-truth model, including I≈1 when the ground truth is isotropic, while its downstream prediction is identically zero. Consequently Eq. (5) cannot detect the harmful shift that Theorem 1 describes, and the paper's claim that isotropy resolves softmax shift-invariance does not follow. In addition, since Eq. (5) defines isotropy directly as the stability of the partition function, the statement that isotropy stabilizes the partition function is definitional rather than a substantive guarantee.","section":"§3, Eq. (5) and Theorem 1"},{"comment":"All empirical claims in Section 5 use the adjusted inter-token cosine similarity ζ'_cos, computed after per-cluster mean subtraction, but the paper provides no derivation connecting ζ'_cos to the partition-function ratio I in Eq. (5) or to the recoverability of logits from probabilities. Without such a bridge, the experiments do not measure the quantity that is claimed to resolve shift-invariance. Moreover, subtracting the cluster mean makes the cosine similarity invariant to translating every vector in a cluster by a constant vector, so Eq. (6) has the same blind spot as Eq. (5) with respect to additive shifts of the kind exploited in Theorem 1.","section":"§4.3, Eq. (6)"},{"comment":"The proof of Theorem 1 constructs arbitrary logit functions {hat z_i(k,l)}, not logits of the form <ψ(k1:Tl), ψ(ki)> used in Eq. (2). As stated, the theorem shows a trivial failure for unconstrained softmax parameterizations, but it does not show that a log-linear student model with matching pretraining loss can have zero downstream prediction. The shift-invariance argument in the text therefore conflates two different model classes, which weakens the motivation for the isotropy condition.","section":"Appendix A and Eq. (2)"},{"comment":"Lemma 1 bounds the spectral norm of the Jacobian of self-attention, and Theorem 2 identifies the matrix Λ that minimizes a related quadratic form, but neither result is connected by any argument to partition-function stability or to the error of the downstream task f(k,l). The text asserts in §4.2 that these results show isotropy 'stabilizing the partition function and preserving the structure needed for reliable numerical downstream task performances,' but this is an unproved leap. Without an explicit inequality linking the Jacobian bound or the eigenvector alignment to I({ψ(k)}) or to f(k,l), these results do not support the claimed performance guarantee.","section":"§4.2, Lemma 1 and Theorem 2"},{"comment":"The quantitative evidence consists of selected examples from individual datasets and models; no aggregate correlation coefficient, confidence interval, or significance test is reported for the relationship between isotropy and NMSE. The claim that isotropy 'strongly influences' or 'directly affects' forecasting performance is therefore not established quantitatively across the full set of 12 real and 10 synthetic datasets, even if the informal visual relationship in the chosen examples is accepted.","section":"§5, Figures 3–8"}],"minor_comments":[{"comment":"There are several typos that should be corrected: 'perfromance' and 'unseed' in Section 3, 'onder' in Section 4.2, 'Cultering' in Appendix D, and 'Transportaion Signal' in Table 4.","section":"Throughout"},{"comment":"The text refers to 'Theorem 3.1' in Section 3, but the theorem is numbered Theorem 1; the Figure 2 caption also spells the model name inconsistently as 'Morai' instead of 'Moirai'.","section":"§3 and Figure 2"},{"comment":"There is an index mismatch in Eq. (4): the KL divergence and entropy terms use p(k_{T_l+1}=j | k_{1:T_l}) on the left-hand side after the equality, while the sums are over i; the indices should be made consistent.","section":"Eq. (4)"},{"comment":"The statement that isotropy in the last layer is 'related to the logits and probabilistic inference' should be justified, since the theoretical quantities in Eq. (5) are defined through the embedding matrix Ψ and no layer-specific argument is given.","section":"§5.1"}],"recommendation":"reject","confidential_remarks":"The paper has a novel framing, but the central theoretical guarantee is circular and the experimental metric is disconnected from it. Fixing the problem would require redefining the structural condition so that it is not invariant to the harmful logit shift, and then formally connecting that condition to Eq. (6) and to downstream error. In my view this goes beyond a routine revision and changes the core contribution of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's empirical survey is genuinely new and useful: five time-series foundation models, 22 datasets, and a consistent look at how a cosine-similarity isotropy measure moves with NMSE across domains, context lengths, and noise. That kind of mapping is a real contribution to people who use LLMs for forecasting. The paper is also clearly written and the appendices give full proofs.\n\nBut the central theory doesn't hold. Eq. 5 defines isotropy as I = min Z / max Z. The global logit shift in Theorem 1 is z_i -> z_i + tau. That shift multiplies every partition function by e^tau, so the ratio I is completely unchanged. A student model with the same pre-training loss and f_hat = 0 can have exactly the same isotropy as the ground-truth model, including I close to 1. So 'isotropy implies stable partition function' is true by definition, but it says nothing about the harmful shift. The paper claims this resolves shift-invariance, and it simply doesn't.\n\nThe experimental metric Eq. 6 has the same blind spot. Mean-subtracted cosine similarity is invariant under translating a whole cluster, so if the harmful shift is realized by moving the context embeddings, the measured isotropy won't catch it. The paper never connects Eq. 5 to Eq. 6, so the experiments may not be measuring the property that supposedly matters.\n\nThe Jacobian bound (Lemma 1) and PCA result (Theorem 2) are imported from Kim et al. and Arora et al., and they bound gradient norm, not downstream error. The experiments are two-point comparisons without error bars or aggregate correlations, so the empirical relationship is suggestive, not established.\n\nThe paper is for readers who want a practical diagnostic for LLM time-series forecasting. The empirical mapping can be useful, but it's presented as a performance guarantee, which is not warranted. I'd send this to review—the empirical part is worth a serious look—but I'd expect heavy revision: either remove the guarantee language or fix the theory so it connects the measured quantity to the claimed property.","headline":"The empirical isotropy–NMSE survey is a genuinely new and useful dataset, but the central theoretical claim is defeated by the paper's own definition: the Eq. 5 ratio is invariant under exactly the global logit shift that Theorem 1 constructs.","tokens_in":17566,"tokens_out":3002,"would_cite":false,"duration_ms":23805,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Isotropy in LLM embeddings is the key to reliable numerical forecasts.","keywords":["isotropy","contextual embeddings","large language models","time series forecasting","softmax shift-invariance","partition function","self-attention geometry","numerical downstream tasks"],"falsifier":"Take a fixed set of LLM hidden states from a forecasting run and compute both the partition-function stability ratio of Eq. 5 and the cluster-adjusted cosine similarity of Eq. 6 on the same last-layer representations; if the two orderings disagree—for example, a model with near-zero cosine similarity but high variance in Z(k,l)—then the measured isotropy is not the quantity that resolves shift-invariance, and the central claim would be falsified. A second check: after measuring isotropy, add a constant to every last-layer logit before applying softmax and run the downstream linear task; if isotropy truly resolves shift-invariance, performance must not change, but if logits were already anchored, the correlation with NMSE is explained by something else.","tokens_in":16494,"feed_emoji":"📈","tokens_out":7122,"duration_ms":55005,"temperature":0.7,"pith_summary":"The paper asks when a language model trained to predict the next token can be trusted to predict numbers, and answers: when the hidden representations are isotropic enough to keep the softmax denominator stable. It argues that without this structure, a student model can shift all logits by a constant, leave the next-token probabilities unchanged, and still drive any downstream numerical task to zero. Isotropy—embedding vectors spread evenly in all directions rather than squeezed into a narrow cone—is presented as the geometric property that prevents such shifts from going unnoticed, because a stable partition function makes logits recoverable from probabilities. Experiments across real and synthetic time series support the claim that stronger measured isotropy in the final layer tracks lower normalized mean squared forecast error. The payoff is a diagnostic: isotropy can be measured on embeddings alone and used to tell when an LLM's numerical predictions carry information.","feed_headline":"Isotropy in LLM embeddings predicts numeric forecasting reliability","feed_subtitle":"Measured on hidden states, cluster-level isotropy tracks forecast error across models and datasets.","key_machinery":"The load-bearing object is the partition function Z(k,l)=\\sum_{i=1}^{|V|} \\exp(\\langle \\psi(k_{1:T_l}), \\psi(k_i)\\rangle), the normalization constant of the softmax in the log-linear model. The paper's isotropy ratio I({\\psi(k)}) = \\min_{\\psi\\in C} Z / \\max_{\\psi\\in C} Z makes 'shift-invariance resolvable' a quantitative claim: if Z barely changes across samples, logits are recoverable from probabilities. The machinery that carries the argument is a spectral analysis of self-attention: Lemma 1 bounds the spectral norm of the Jacobian of the attention map, and Theorem 2 shows that the minimizing Λ is a projection onto the m largest eigenvectors of the embedding correlation matrix, so attention naturally creates low-dimensional local structure. Empirically, isotropy is measured as the adjusted inter-token cosine similarity after per-cluster mean subtraction, motivated by the observation that embeddings form separated cluster islands and a global cosine would be dominated by inter-cluster distances. The whole chain is: stable partition function, isotropy, cluster-adjusted cosine similarity, and forecasting error.","core_discovery":"The paper's central claim is that the hidden representations of an LLM need a specific geometric structure—isotropy in the contextual embedding space—to make the softmax-based next-word prediction usable for numerical downstream tasks. The reasoning runs through three steps. First, Theorem 1 shows that for any bounded ground-truth logits, one can add a constant shift to all logits so that the student distribution is identical to the ground truth while the downstream score f(k,l) becomes zero; without structural constraints, pre-training loss cannot certify numerical performance. Second, stability of the partition function Z(k,l) is identified with isotropy, since I({ψ(k)}) = min Z / max Z approaches 1 exactly when the partition function is nearly constant. Third, a Jacobian bound on self-attention and a spectral argument show that self-attention implicitly aligns with the top eigenvectors of the embedding correlation matrix, inducing cluster-level isotropy rather than global isotropy. The paper then measures isotropy as the adjusted inter-token cosine similarity after subtracting each cluster's mean, and reports that across datasets, contexts, and noise levels, higher isotropy in the last layer is associated with lower forecasting error.","pith_inferences":["Editorial inference: the paper leaves implicit that isotropy is a necessary but not sufficient condition; a stable partition function fixes shift-ambiguity, but does not by itself guarantee that the logits contain the signal the downstream task needs. A testable extension would be to add a constant to all logits after measuring isotropy and check whether linear readouts degrade in spite of unchang","Editorial inference: the theoretical ratio in Eq. 5 and the measured cluster cosine similarity in Eq. 6 are not formally connected in the text; linking them with a quantitative inequality would turn the empirical correlation into a guarantee. One can compute both on the same hidden states and check whether the rank ordering matches.","Editorial inference: if isotropy is the right control variable, then interventions that increase it—whitening last-layer embeddings, increasing context length, or denoising inputs—should improve downstream NMSE, a direct experiment the paper does not run.","Editorial inference: the result also suggests a calibration story: when the partition function is stable, probabilities and logits carry the same information, which is precisely the condition under which linear probes on embeddings should work for forecasting."],"forward_implications":["Isotropy becomes a cheap diagnostic: measure it on final-layer embeddings of any LLM before deploying it for time series forecasting, and use it to flag contexts where numerical outputs are untrustworthy.","Context length and noise level act through isotropy: shortening context or adding Gaussian noise lowers measured isotropy, which the paper links to higher NMSE, giving a principled way to choose context windows instead of trial and error.","Self-attention's implicit alignment with dominant eigenvectors means architecture and tokenization choices that preserve local isotropy will also protect downstream numerical reliability.","A model with low pre-training loss is not enough: Theorem 1 implies that without structural isotropy, logits can be useless for numerical tasks even when probabilities match the data distribution.","Because isotropy varies by dataset and model, no single LLM is universally reliable; reliability is a property of the model-data pair, and isotropy measures that pair."],"supporting_citations":[{"why":"Supplies the log-linear model and the link between embedding isotropy and the partition function used in Eqs. 2 and 5.","marker":"Arora et al. [2016]"},{"why":"Provides the isotropy notion and the mean-subtraction/all-but-top view adopted for the adjusted cosine similarity in Eq. 6.","marker":"Mu and Viswanath [2018]"},{"why":"Gives the self-attention Lipschitz/Jacobian analysis that Lemma 1 adapts to bound the gradient norm.","marker":"Kim et al. [2021]"},{"why":"Establishes the representation-property framework connecting pre-training loss to downstream tasks, including the logit-shift argument behind Theorem 1.","marker":"Wu et al. [2023]"},{"why":"Defines the time-series tokenization and KernelSynth synthetic data generator used in the experiments.","marker":"Ansari et al. [2024]"},{"why":"Motivates cluster-level isotropy and effective dimension by showing clustered structure in contextual embedding spaces.","marker":"Cai et al. [2021]"},{"why":"Supports the low-effective-dimension observation with evidence that contextual embeddings occupy a narrow cone.","marker":"Ethayarajh [2019]"},{"why":"Provides the silhouette score used to choose the number of clusters for the isotropy measurement.","marker":"Rousseeuw [1987]"}],"fun_headline_variants":["Isotropy predicts LLM numeric forecasting accuracy","LLMs need isotropy to forecast numbers reliably","When isotropy boosts LLM next-word prediction for numbers","Isotropy is key to LLMs handling numerical domains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the isotropy measured by cluster-adjusted cosine similarity (Eq. 6) is the same property as the partition-function stability ratio defined in Eq. 5; if that bridge fails, the empirical correlation does not test the theory.","fun_headline_variants_meta":{"raw":{"variants":["Isotropy predicts LLM numeric forecasting accuracy","LLMs need isotropy to forecast numbers reliably","When isotropy boosts LLM next-word prediction for numbers","Isotropy is key to LLMs handling numerical domains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3103,"prompt_tokens":1078,"completion_tokens":2025,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":1962}},"tokens_in":694,"tokens_out":2025,"duration_ms":13489,"temperature":1.0,"reasoning_tokens":1962,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:04:55.916798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of LLM hidden states from a forecasting run and compute both the partition-function stability ratio of Eq. 5 and the cluster-adjusted cosine similarity of Eq. 6 on the same last-layer representations; if the two orderings disagree—for example, a model with near-zero cosine similarity but high variance in Z(k,l)—then the measured isotropy is not the quantity that resolves shift-invariance, and the central claim would be falsified. A second check: after measuring isotropy, add a constant to every last-layer logit before applying softmax and run the downstream linear task; if isotropy truly resolves shift-invariance, performance must not change, but if logits were already anchored, the correlation with NMSE is explained by something else.","supporting_citations":[],"review_version":1}