{"id":"193dd9bc-ec6f-489f-973f-5d530217e740","arxiv_id":"2608.03015","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"As context grows, LLM internal representations of numerical sequences become more clearly separated by input complexity: simple inputs yield smoother, more globally connected token graphs, while chaotic inputs yield more localized graphs with broader spectral support.","lead":"This paper uses graph signal processing to look inside large language models while they predict number sequences. It finds that longer prompts make the models' internal token graphs and hidden states systematically different for simple versus chaotic inputs, and this pattern appears across several model families.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aggregated attention graph may conflate raw token statistics with complexity ordering; an explicit null model over quantized trajectories is needed.","rationale":"The reader identified the aggregation/directionality of the attention graph as the weakest assumption. That is a real concern, but I find a more immediately decisive issue: the construction pairs attention graphs with hidden-state signals at the same layer, and both objects are computed from the same quantized trajectory. The paper's controlled input suite deliberately spans a complexity gradient that correlates strongly with token-level statistics (state-distribution entropy, number of distinct states, recurrence patterns). The non-graph baselines in Appendix H (DCT positional HFER/SE, attention Gini) already show that broad context-dependent trends appear even without the graph construction, which the paper interprets as evidence that the graph-spectral formulation adds separation. However, those baselines still consume the ordered sequence and real hidden states; they do not control for the marginal statistics of the quantized tokens or for token-order scrambling. A shuffled-token control is the minimal test of whether the claimed internal dynamical-complexity ordering is a property of the model's sequence processing or a property of the token statistics alone. The paper provides public code, so this test is concretely runnable. I do not think this concern invalidates the paper; the central claim remains plausible, and the existing robustness checks are thoughtful. But the conclusion says 'systematic, context-dependent internal signatures ... conserved across model families,' and a shuffled-input null is a standard control in representation analyses of LLMs. Without it, the conditional verdict is appropriate: the finding should be accepted conditional on the null-model check, and the paper should add the control and a significance statement. I partially agree with the reader because the aggregation assumption is also worth testing (e.g., per-head or directed variants), but the more load-bearing and cheaper-to-test issue is the lack of a shuffled-token/statistics null model.","tokens_in":28594,"tokens_out":2169,"duration_ms":21612,"concrete_test":"Run the exact Section 4.1 pipeline (Llama-3.2-3B, same 20 context lengths, same diagnostics λ2, HFER, exp(SE)) on three control conditions: (1) for each realization, shuffle the serialized tokens within each context length while preserving the token multiset (destroying trajectory order); (2) shuffle the 1000-state trajectory values before quantization, preserving only the marginal distribution of states; (3) replace the attention matrix with a uniform attention baseline while keeping hidden states unchanged. If the complexity-ordered separation in Fig. 3 (especially the chaotic-vs-periodic λ2/HFER separation and the turning-point ordering) persists under condition (1) or (2), then the claimed dynamical-complexity signatures are largely inherited from token/state statistics rather than from in-context sequence processing, and the central claim must be weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that LLM internal representations are systematically ordered by input dynamical complexity, not merely by prediction error (Section 4.1, Fig. 3, conclusion). The weakest load-bearing premise is that the graph-spectral diagnostics measure organization induced by the LLM rather than inherited almost unchanged from the raw numerical input. The paper's key controls (Appendix H) compare graph-spectral diagnostics against DCT positional baselines and attention-Gini, but they do not test the simplest possible null model: feed the serialized quantized trajectories through a trivial attention model or through a token-position-independent baseline that preserves only token identity statistics (e.g., a context-free 'bag of tokens' spectral graph). The inputs are curated so that complexity correlates with state-distribution entropy (Table 1) and with quantized token statistics: chaotic trajectories visit many more distinct three-digit strings than constant/low-period inputs. Since nodes are tokens and signals are hidden states entering the layer, a model whose attention weights and hidden states were driven primarily by the distinct token types and their co-occurrence frequencies would reproduce the same qualitative λ2/HFER/exp(SE) ordering without any genuine dynamical-complexity processing. Specifically, a randomized baseline that permutes token identities within each context or replaces the trajectory with a shuffled version of the same multiset of tokens would test whether the ordering is an artifact of token statistics rather than sequence organization. The paper's own Appendix H control (DCT-based positional HFER/SE) is not a sufficient null: it still uses the actual ordered sequence and actual hidden states, so it cannot rule out that the hidden-state statistics alone (without any attention-graph organization) explain the ordering. Moreover, the visualizations (Fig.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a graph-signal-processing framework for analyzing LLM internal representations during numerical in-context learning (ICL). At each transformer layer, the symmetrized, head-averaged post-softmax attention matrix defines a weighted undirected graph over tokens, and the hidden states entering the layer define vector-valued signals on the graph nodes. The authors compute three layer-averaged diagnostics—normalized Fiedler value λ2, high-frequency energy ratio (HFER), and exponentiated spectral entropy exp(SE)—and pair them with Fruchterman–Reingold token-graph visualizations. Using a controlled suite of constant, periodic (logistic map, periods 2–64), and chaotic (logistic r = 3.9, 4.0; Lorenz) inputs serialized as comma-delimited three-digit tokens, they report that, as context length grows from 100 to 2000 tokens, chaotic inputs develop lower λ2, higher HFER, and broader exp(SE) than non-chaotic inputs, with periodic inputs ordered by period length. The trends are reproduced across Llama 3 (1B/3B/8B, base and instruct), Phi-4, and SmolLM3, and are robust to HFER cutoffs, layers, and layout choices. The paper concludes that internal representations are systematically, context-dependently organized by input dynamical complexity in a way that output-level extrapolation error alone does not reveal.","tokens_in":28783,"tokens_out":14294,"duration_ms":147194,"significance":"The manuscript is honest, careful, and unusually well controlled on the descriptive side. The code and data are publicly released; the two free parameters (HFER cutoff, display threshold) are robustness-tested (Appendices D.1 and E); the layer-averaged results are cross-checked against final-layer and per-layer analyses (Appendix D.2); and the cross-model conservation across Llama 3, Phi-4, and SmolLM3 is a genuine strength. The observation that representation-level diagnostics separate input families whose output-level errors remain tightly clustered (Fig. 5 versus Fig. 3) is a substantive, falsifiable claim, and the Limitations section appropriately disclaims causal mechanisms. However, the central interpretive step—attributing the ordering to input dynamical complexity—is currently vulnerable to a token-statistics confound, and the fine-grained ordering claims lack inferential statistics. Both are fixable within the manuscript's scope; if the requested null-model and per-head or directed-attention checks confirm the ordering, this would be a valuable contribution to the mechanistic-interpretability and time-series ICL literatures.","major_comments":[{"comment":"The fine-grained claims in Section 4.1 rest on visual inspection of curves with ±1 SD shading over 20 realizations per family, with no significance tests or effect sizes. Specifically, the 'ordered turning points' claim (periods 32 and 64 turn upward at N = 300 and N = 400) and the period-ordered HFER ladder are load-bearing for the paper's central claim that periodic inputs are ordered by period length, and multiple-comparison concerns arise across 10 families and 20 context lengths. I recommend permutation tests comparing chaotic versus non-chaotic groups at each N, a standardized effect-size measure (for example, Cohen's d or an AUC) for the λ2/HFER/exp(SE) separations, and a rank-correlation test between period length and diagnostic values at fixed N to substantiate the ladder.","section":"§4.1, Fig. 3"}],"minor_comments":[{"comment":"The period-64 narrative—a clique-like subnetwork concentrated in the first roughly 128 tokens that disappears once recurrence is revealed—is based on representative visualizations; quantifying this across the 20 realizations (for example, the local clustering coefficient or conductance of the detected cluster as a function of token position) would make the claim reproducible and would also justify the 'clique-like' terminology used for the chaotic inputs.","section":"§4.2, Fig. 4"},{"comment":"Node colors are obtained by per-input min–max scaling of the leading three uncentered SVD coordinates (Appendix E), so colors are not comparable across panels; this caveat should appear in the main-text caption to avoid over-interpreting cross-input color differences.","section":"§3.3, Fig. 2"},{"comment":"The claimed turning points for periods 32 and 64 should be marked on the figure or defined by an explicit criterion, since they are load-bearing for the interpretation that the model needs progressively more context to resolve longer-period dynamics and are currently identified by eye.","section":"§4.1, Fig. 3(a)"},{"comment":"There are several spacing and subscripting artifacts in the displayed appendix equations (for example, 'rX k=1' in Appendix E and missing spacing around operators in Appendix B); a proofreading pass over the appendices would improve readability.","section":"Appendices B, E, H"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper is a solid empirical study within the journal's scope, and rejection would be disproportionate because the main gap—the missing token-shuffle null model—is straightforward to close with the released code. My recommendation of major_revision is driven by two issues: (i) the input design confounds dynamical complexity with the diversity of distinct three-digit token types, and the Appendix H controls use intact sequences and hence cannot falsify a token-statistics explanation; (ii) the headline separations are presented without significance tests or effect sizes. I would expect one focused revision round to resolve both. I also note that the Appendix H DCT baselines already show the broad trends in hidden states alone; the authors interpret this as validating their framework, but it actually underscores the need for the shuffle control."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this paper if you care about what LLMs do internally during numerical in-context learning. The novel piece is the systematic finding that, as context grows, graph-spectral diagnostics over attention-induced token graphs separate chaotic from periodic inputs and order periodic families by period, and that this ordering is conserved across Llama, Phi, and SmolLM3. The math is standard graph signal processing, applied correctly, and the robustness checks are thoughtful: alternative HFER cutoffs, low-end eigenvalue spectra, layerwise checks, layout initialization, edge display thresholds, and non-graph controls all support the qualitative trends. The paper is honest about its mechanistic limits. That is real value for a mesoscale interpretability toolkit.\n\nThe main soft spot, which the reader's report flags, is the graph construction: symmetrizing and head-averaging attention discards directional information, and the paper does not test per-head or directed variants. I think that is a moderate concern, not a fatal one, because the diagnostics are explicitly mesoscale summaries and the attention-only Gini control partially addresses it. The stress-test note goes further and asks for a shuffled-token or bag-of-tokens null model, arguing that complexity correlates with token multiset entropy and that the ordering could be inherited from token statistics rather than genuine dynamical processing. That is a legitimate and missing control. The existing Appendix H baselines still use ordered sequences and real hidden states, so they cannot rule out a token-identity artifact. A simple permutation test of token order within each context would settle whether the spectral signatures require sequence organization or just token frequencies. I would not call the paper's central claim misleading without that test, but I would want it before treating the ordering as evidence about dynamical-complexity tracking rather than input statistics.\n\nA smaller issue: the separations in Fig. 3 are shown with means and standard deviations but no significance tests or effect sizes. For some families, especially the mid-period ones, the shaded bands overlap; a reader cannot tell how robust the ladder is. That is easy to fix.\n\nWho is this for? Anyone doing interpretability of numerical ICL or using graph-spectral diagnostics on transformer internals. It deserves a serious referee: the question is meaningful, the execution is clean, and the missing null model is a request for additional evidence, not a refutation. I would send it to review with a request for a shuffled-token baseline and significance testing.","headline":"A solid, clearly written empirical study of internal graph-spectral signatures in numerical ICL; the main missing control is a shuffled-token/null baseline, but the core descriptive finding holds.","tokens_in":29420,"tokens_out":2126,"would_cite":true,"duration_ms":25645,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"As context grows, LLM attention graphs and hidden-state signals sort numerical inputs by dynamical complexity, from constant and periodic to chaotic.","keywords":["in-context learning","graph signal processing","graph Laplacian spectrum","attention graphs","dynamical complexity","logistic map","LLM interpretability","spectral entropy"],"falsifier":"A concrete check: recompute the three diagnostics per attention head and with a directed Laplacian rather than the symmetrized, head-averaged graph. If the chaotic-versus-periodic separation in the normalized Fiedler value, HFER, and spectral support disappears under either variant, the reported ordering is an artifact of the aggregation recipe rather than a property of the model. A second check is a permutation test that scrambles hidden states across nodes while keeping the attention graph fixed; if HFER differences persist after scrambling, the spectral support is not actually tied to the token graph.","tokens_in":28382,"feed_emoji":"📈","tokens_out":8866,"duration_ms":82296,"temperature":0.7,"pith_summary":"This paper claims that during numerical in-context learning (ICL), the internal organization of a large language model tracks the dynamical complexity of the input sequence, not just the output error. The authors treat attention weights as defining a weighted graph over tokens and hidden states as signals on that graph, then measure three spectral quantities: the normalized Fiedler value for global connectivity, the high-frequency energy ratio (HFER) for signal roughness, and the exponentiated spectral entropy for effective spectral support. They find that as context length increases, simpler inputs (constants and short-period sequences) produce more globally connected attention graphs and smoother, spectrally concentrated hidden states, while chaotic inputs produce more localized graphs with higher high-frequency energy and broader spectral support; periodic inputs fall in between, ordered by period length. The ordering holds across Llama 3.2 1B/3B/8B, Phi-4, and SmolLM3, so the authors conclude the signatures are conserved across model families and become sharper with context.","feed_headline":"Attention graphs reveal how LLMs order numbers by chaos","feed_subtitle":"Simple sequences get smoother attention graphs; chaotic ones stay fragmented — seen across model families.","key_machinery":"The central construction is the layerwise token graph: at each transformer layer, each head's post-softmax attention matrix is symmetrized and averaged across heads to form a weighted, undirected adjacency matrix over token nodes, and the hidden states entering that layer form vector-valued signals on those nodes. The normalized graph Laplacian's eigenvectors provide a graph Fourier basis, and the signal's projection onto that basis yields the three diagnostics: the normalized Fiedler value (second-smallest normalized Laplacian eigenvalue) measures global connectivity, HFER measures the fraction of hidden-state energy in the upper half of graph frequencies, and exponentiated spectral entropy measures the effective number of equally weighted modes. Qualitative token-graph visualizations use a force-directed layout driven by attention weights, with node colors encoding the leading three SVD components of hidden states, so the same organization can be seen as clique-like subnetworks that appear only for chaotic inputs or for long-period inputs before enough context has resolved their recurrence.","core_discovery":"The paper's central claim is that LLM internal representations during numerical ICL exhibit systematic, context-dependent structure organized by input dynamical complexity. Concretely, at longer contexts the layer-averaged diagnostics separate chaotic logistic and Lorenz trajectories from constant and periodic trajectories: chaotic inputs have lower normalized Fiedler value (weaker global attention-graph connectivity), higher HFER (hidden-state energy shifted toward high graph frequencies), and larger effective spectral support (energy spread across more graph Fourier modes). Among periodic logistic inputs the separation is graded by period, and the turning points for periods 32 and 64 suggest the model needs progressively more context to recognize longer recurrences. The authors also report that graph-spectral diagnostics separate input families whose output-level extrapolation errors are tightly clustered, indicating the internal signatures are not simply a reflection of prediction error.","pith_inferences":["A natural extension the paper leaves open is causal testing: ablating or amplifying particular graph-frequency bands of hidden states could determine whether the smoother low-frequency signals for simple inputs are actually used for next-state prediction.","The preliminary natural-language and code results suggest numerical sequences occupy a distinct spectral region (more globally connected, lower HFER, narrower support); testing whether this separation tracks token-level predictability or training-data exposure would be a concrete follow-up.","The period-ordered turning points in the Fiedler value hint at an internal 'context needed to identify regime' metric, which could be used to predict when an LLM's extrapolation will become reliable before consulting outputs.","Per-head and directed attention analyses would show whether the aggregation step hides meaningful directional structure; if the signatures survive at the per-head level, they would be a stronger invariant of the model."],"forward_implications":["Longer context makes LLM internal organization more clearly reflect input dynamical complexity, so representation-level measurements complement output-level prediction error as a way to study numerical ICL.","For periodic inputs, the graph-spectral diagnostics produce a period-ordered ladder, with higher-period sequences initially behaving like chaotic ones until enough context reveals the recurrence.","Because the trends persist across Llama 3.2, Llama 3.1, Phi-4, and SmolLM3, the organization appears to be a general feature of pretrained LLMs rather than a quirk of one architecture.","Instruction-tuned variants show weaker separation between high-period and chaotic inputs than base models, suggesting post-training can blur these internal dynamical-complexity signatures.","Attention-only and hidden-state-only controls reproduce broad trends but with less fine-grained separation, implying the joint graph-signal view carries information that neither signal alone provides."],"supporting_citations":[{"why":"Supplies the three spectral diagnostics (normalized Fiedler value, high-frequency energy ratio, spectral entropy) that the paper adapts to numerical ICL.","marker":"Noël (2026)"},{"why":"Defines algebraic connectivity, the basis for interpreting the Fiedler value as a measure of global graph connectivity.","marker":"Fiedler (1973)"},{"why":"Provides the normalized Laplacian and its spectral theory, which the paper uses to define the normalized Fiedler value.","marker":"Chung (1997)"},{"why":"Establishes the graph Fourier transform interpretation that lets hidden states be analyzed as signals on the attention-induced graph.","marker":"Shuman et al. (2013)"},{"why":"Demonstrates zero-shot time-series forecasting with LLMs, the output-level capability whose internal organization this paper studies.","marker":"Gruver et al. (2023)"},{"why":"Prior work on LLMs learning governing principles of dynamical systems that motivates the input suite and serialization choices.","marker":"Liu et al. (2024)"},{"why":"Provides prior internal-representation evidence of curved belief manifolds, motivating joint hidden-state analysis and the Gaussian-switch inputs.","marker":"Sarfati et al. (2026)"},{"why":"Introduces the logistic map whose period-doubling route supplies the controlled complexity ladder spanning periods 2 through 64 and chaos.","marker":"May (1976)"},{"why":"Supplies the continuous chaotic Lorenz system used as an additional chaotic input family.","marker":"Lorenz (1963)"}],"fun_headline_variants":["Attention graphs expose LLMs' internal coding of numerical chaos","Graph spectra show LLMs separate simple from chaotic sequences","LLM internal structure mirrors dynamical complexity of inputs","Smoother attention graphs for simpler numbers in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the symmetrized, head-averaged attention matrix faithfully represents the token graph, so the spectral diagnostics measure genuine internal organization rather than artifacts of averaging away directional and per-head attention structure.","fun_headline_variants_meta":{"raw":{"variants":["Attention graphs expose LLMs' internal coding of numerical chaos","Graph spectra show LLMs separate simple from chaotic sequences","LLM internal structure mirrors dynamical complexity of inputs","Smoother attention graphs for simpler numbers in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1320,"prompt_tokens":870,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":386}},"tokens_in":486,"tokens_out":450,"duration_ms":6156,"temperature":1.0,"reasoning_tokens":386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:15:41.339549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: recompute the three diagnostics per attention head and with a directed Laplacian rather than the symmetrized, head-averaged graph. If the chaotic-versus-periodic separation in the normalized Fiedler value, HFER, and spectral support disappears under either variant, the reported ordering is an artifact of the aggregation recipe rather than a property of the model. A second check is a permutation test that scrambles hidden states across nodes while keeping the attention graph fixed; if HFER differences persist after scrambling, the spectral support is not actually tied to the token graph.","supporting_citations":[{"cited_title":"and Boull","cited_arxiv_id":null,"evidence_quote":"Prior work on LLMs learning governing principles of dynamical systems that motivates the input suite and serialization choices."}],"review_version":1}