{"id":"dbe5d54b-c39c-46d8-b776-24683f080d6a","arxiv_id":"2608.10986","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Self-feeding language model probes mix construction-determined and model-determined signals, and a two-way variation test separates the two kinds of readings.","lead":"This paper studies what iterated self-feeding probes of language models actually measure, using a ring of tokens that the model resamples from its own predictions. It shows that some sharp, reproducible signals come from the probe setup rather than the model, and offers a simple variation test to tell the two apart.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The symmetric-window conditional p_r(x_i | x_{i±r}) is never defined for autoregressive models in §2; the paper only states it is 'imposed' on them. All principal readings (Pythia, GPT-2, Granite) depend on this unspecified operation.","rationale":"The reader identified the exact-zero null as the weakest assumption, but the paper provides a strong validation of that null through the Domany–Kinzel identity: the CRN damage field matches an independent prediction cell-for-cell (0 mismatches across five p1 values, 4096-cell ring, 1500 steps, three seeds). That test exercises the same shared-uniform sampling loop used for language models, so on the backend actually used, the exact-zero property is empirically supported. The symmetric-window conditional for autoregressive models is different: it is a definitional gap in the construction itself. The central claim is that self-feeding probes mix construction-determined and model-determined quantities and that a variation test separates them. If the autoregressive construction is not well-defined, then the readings used to demonstrate the discriminator (the manufactured transition, the developmental crossing, the attractor share ordering) may be artifacts of an unspecified approximation rather than properties of the stated probe. This is more load-bearing because it challenges the identity of what was measured, not just the numerical reproducibility on untested backends. The paper explicitly flags other limitations but never mentions this ambiguity, and the sentence 'imposed on an autoregressive one' is the only explanation given. The concrete test is feasible: the code is released, and inspecting the sampling path would settle whether a legitimate bidirectional condition is computed or whether the paper's construction differs from its own definition. Until that is clarified, the empirical demonstrations on autoregressive models should be treated as contingent on the implementation detail. Hence the verdict should shift from a confident ACCEPT to CONDITIONAL: the scientific content is strong, but the paper must specify or correct the autoregressive symmetric-window computation before the central examples can be taken at face value.","tokens_in":12284,"tokens_out":13633,"duration_ms":139288,"concrete_test":"Inspect the released repository (github.com/nicoveraz/token-lattice-ca), specifically the sampling function for autoregressive models. Verify how p_r(x_i | x_{i−r},…,x_{i+r}) is computed: is there a genuine bidirectional condition (e.g., a masked-LM-style forward pass, a permutation-based factorization, or a true infilling method) or is it approximated using causal logits only? If the former, the concern is resolved. If the latter, re-run a minimal λ_ca measurement (e.g., Pythia-410m at T=0.5, r=2) with the true bidirectional conditional (using a masked LM) and compare against the reported value; a large discrepancy would indicate the autoregressive construction deviates from the stated definition.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2 defines the update as sampling from p_r(x_i | x_{i−r},…,x_{i−1}, x_{i+1},…,x_{i+r}) with the centre masked, then says this is 'native to a masked language model and is imposed on an autoregressive one'. For a causal LM, such a symmetric conditional is not defined: the model computes p(x_i | x_{<i}) only. The paper gives no mechanism — no bidirectional attention variant, no permutation scheme, no infilling head, no approximation with its error analysis — by which the autoregressive models (Pythia, GPT-2, Granite) yield the claimed symmetric-window probability. This is not a backend-reproducibility issue but a definitional one: if the implementation instead uses left-context-only logits or a heuristic combination, then λ_ca, the attractor share, and the manufactured/developmental transitions are measurements of that approximation, not of the stated construction. The exact-zero null, by contrast, is empirically validated on the used backend by the bit-exact Domany–Kinzel reproduction (§3, Rung 1: 0 mismatching cells), so the reader's main concern about nondeterministic backends is real but less fundamental: it affects generality, not the definition of the measured quantity. The symmetric-window gap affects all autoregressive-model results at the level of what is being measured. The paper should specify the exact computation, or point to the precise code path, before the discriminator's demonstrations can be taken as evidence about the construction.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies iterated self-feeding probes of language models by constructing a ring of token cells resampled in place from a windowed conditional, with two nearly identical rings advanced under common random numbers. It claims that the exact-zero null makes damage spreading measurable, that a sharp low-temperature absorbing-state transition is a property of the probe rather than of any language model, and that a variation test can separate construction-determined from model-determined readings. The instrument is calibrated by bit-exact reproduction on the Domany-Kinzel automaton, and the paper reports several retracted verdicts and estimator-gating guards. All code and per-run result files are released.","tokens_in":12603,"tokens_out":4191,"duration_ms":42154,"significance":"If the main claims hold, the paper is a valuable methodological contribution: it demonstrates that a sharp, reproducible phase transition can be an artifact of the probe, and it offers a simple, transferable test for classifying readings of self-feeding loops. The bit-exact Domany-Kinzel reproduction, the explicit retractions, and the reproducible package are concrete strengths that support the paper's central message. The universal wording in places goes beyond the finite evidence, but the discrimination test itself does not depend on that universalization.","major_comments":[{"comment":"The symmetric-window conditional in Eq. (1) is well-defined for masked language models, but no mechanism is given by which autoregressive models (Pythia, GPT-2, Granite) provide p_r(x_i | x_{i−r}, …, x_{i−1}, x_{i+1}, …, x_{i+r}) with the centre masked. A causal LM computes only left-context probabilities, so the implementation must use some approximation such as a bidirectional attention variant, a permutation scheme, an infilling head, or a heuristic combination. This is not a reproducibility footnote: if the autoregressive numbers come from a left-context-only or heuristic computation, then λ_ca, the developmental transition, and the attractor-share results are measurements of that approximation, not of the stated construction. The paper should specify the exact computation or identify the precise code path before the discriminator demonstrations can be taken as evidence about the construction.","section":"§2, Eq. (1)"},{"comment":"The exact-zero null is stated as holding on every backend because inverse-CDF sampling against a shared uniform makes agreement exact. That guarantee is not valid for every possible backend: nondeterministic sampling kernels, approximate sampling, or thread-level RNG divergence would break exact agreement. The bit-exact Domany-Kinzel reproduction in Rung 1 validates the null on the used backend, so the central measurements survive, but the universal claim should be scoped to the validated backend and to the code path that implements the shared uniform stream.","section":"§2, exact-zero null"},{"comment":"The term 'model-invariant' for λ_ca(r) is stronger than the evidence supports. The demonstration covers 19 models and two scale ladders, which is a substantial sample, but it does not establish invariance across models in general, especially because §5.2 reports a cross-model spread of 0.051 against a range of 0.122–0.804 produced by varying radius and temperature alone. I recommend saying 'invariant across the tested models and constructions' and giving a quantitative criterion for when a spread counts as 'not moving' relative to the construction-driven range.","section":"§5.1"}],"minor_comments":[{"comment":"The notation p_r(x_i | x_{i+-r}) in the abstract is nonstandard and ambiguous; the intended symmetric window appears in Eq. (1), and the abstract should use the same notation.","section":"Abstract"},{"comment":"The sentence 'Monochrome by marker and dash throughout' is unclear; please state explicitly how the five arms are distinguished if color is not used.","section":"Figure 3 caption"},{"comment":"The symbol T* and the phrase 'greedy degeneration' are used without definition, which makes the predictive-result paragraph hard to evaluate; please define both before the claims that rely on them.","section":"§7"},{"comment":"The sentence 'its model ordering is seed-stable at 0.848 against 0.030' is ambiguous; please specify whether 0.848 is a correlation, an agreement rate, or another statistic, and state the comparison being made.","section":"§5.2"},{"comment":"The paper says 'ten models spanning six families and four architecture classes' but does not list them; a small table or appendix listing the models, families, and parameter counts would make the invariance and ordering claims easier to check.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually honest and the methodological core is strong. I would be comfortable with acceptance after the autoregressive conditional is specified precisely, the 'every backend' and 'model-invariant' claims are scoped to the validated evidence, and the minor clarity issues are fixed. The symmetric-window gap is load-bearing for the autoregressive demonstrations, but it appears fixable by pointing to the exact code path or by re-scoping the claims if the implementation uses a heuristic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is worth your time. The paper makes a sharp, useful point: iterated self-feeding probes mix construction-determined and model-determined quantities in readings that look identical, and the only way to tell them apart is to vary one factor while holding the other fixed. That discriminator is real, and the worked example — a phase transition measured to three decimal places that belongs entirely to the probe — is a convincing demonstration that the problem is not hypothetical. The common-random-numbers coupling and the exact-zero null are genuinely new in this setting, and the calibration on the Domany–Kinzel automaton (bit-exact, zero mismatching cells) is as strong a validation as this kind of instrument can get.\n\nThe paper also does something rare in this literature: it reports four retracted verdicts, each caught by a known-answer system rather than by review, and it ships the guards as a package. That honesty is not decorative; it supports the central claim that estimator discipline matters more than the specific numbers.\n\nNow the soft spots. The stress-test concern is legitimate and should be fixed before publication: Section 2 defines the update as a symmetric-window conditional p_r(x_i | x_{i±r}), says this is \"imposed\" on autoregressive models, but never specifies the mechanism. Later the paper mentions an \"autoregressive (causal, left-window) construction,\" which suggests the implementation used left-context-only logits, but the main text never reconciles the two. That is a definitional gap, not just a reproducibility detail, because the autoregressive results (Pythia, GPT-2, Granite) are measurements of whatever approximation was actually used. The code is public, so a careful reader can chase it down, but the paper should say exactly what was computed. This is a moderate revision, not a fatal flaw.\n\nThe model-invariance claim across 19 models is also stronger than the evidence: finite sample, and the cross-model spread is small but not zero. That is acknowledged in the limits, so it is a scope caveat rather than an error.\n\nOverall: the central argument holds, the method is validated, and the write-up is unusually candid. With the autoregressive-window gap specified, this is a solid contribution. I would send it to a serious referee.\n\nRecommendation: engage with it — send for peer review, with emphasis on the specification of the autoregressive construction.","headline":"A self-critical methods paper that earns its central claim: CRN coupling plus a variation test cleanly separates construction-determined from model-determined readings, and the reproducible artifacts make the soft spots fixable.","tokens_in":13082,"tokens_out":1531,"would_cite":true,"duration_ms":16096,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Iterated self-feeding probes of language models read probe and model at once; a variation test separates which is which.","keywords":["self-feeding probes","damage spreading","common random numbers","Lyapunov exponent","language model dynamics","Glauber dynamics","directed percolation","model attribution"],"falsifier":"Advance two undamaged rings under common random numbers on a backend with nondeterministic or approximate sampling; if their divergence is ever nonzero, the exact-zero null fails and all damage-spreading readings are contaminated.","tokens_in":12102,"feed_emoji":"🔄","tokens_out":7636,"duration_ms":70824,"temperature":0.7,"pith_summary":"Self-feeding a language model its own output is now routine, but what such a probe measures has stayed unclear. This paper argues that an iterated probe reads the model and the probe at once, in quantities that carry the same units and apparent precision but answer to different causes. To make the question sharp it builds a ring of token cells resampled in place by the model's own windowed conditional, and couples two rings that differ in one token under common random numbers, making damage spreading measurable. The central result is a discriminator: hold the construction fixed and vary the model, or hold the model fixed and vary the construction, and see which readings move. Applying it, the paper shows a phase transition it measured to three decimal places belongs to the probe, while a developmental transition in the same instrument belongs to the model.","feed_headline":"One variation test splits self-feeding LLM readings into model and probe","feed_subtitle":"A ring-lattice experiment shows a measured phase transition can be pure probe, and how to tell the model's readings apart.","key_machinery":"The load-bearing object is the common-random-numbers coupling of two rings. Each ring is a set of $N$ token cells resampled in place from the model's windowed conditional $p_r(x_i \\mid x_{i\\pm r})$ at temperature $T$; two rings are identical except for one flipped token and are advanced with the same stream of uniform variates and the same visit order. Because inverse-CDF sampling against a shared uniform makes identical windows draw identical tokens, undamaged twins diverge by exactly zero, so every difference is attributable to the injected flip. That exact-zero null turns the construction into a damage-spreading instrument: the exponential growth rate of the damage gives the Lyapunov exponent $\\lambda_{\\mathrm{ca}}$, its saturation gives a damping length, and its spatial extent gives a light cone. The discriminator of Section 5 uses the same idea one level up: vary the construction with the model fixed, or the model with the construction fixed, and classify each reading by whether it moves.","core_discovery":"The paper's central claim is that iterated self-feeding probes do not have a single object: they mix construction-determined quantities with model-determined ones in readings that look alike and cannot be told apart by inspection. In the ring construction, the damage light cone is kinematic, its extent set by the update window rather than by the model, and the radius scaling of the token-space Lyapunov exponent $\\lambda_{\\mathrm{ca}}(r)$ is model-invariant across 19 models and two scale ladders spanning 70×. Other readings genuinely track the model: $\\lambda_{\\mathrm{ca}}$ crosses zero at a reproducible point in training, and the attractor share ranks models consistently however the lattice is built. The paper's test for separating the two kinds is a manipulation: vary one factor with the other held fixed and watch which readings move. By that test, a sharply measured absorbing-state phase transition with directed-percolation exponents at a common $T_c \\in [0.4343, 0.4391]$ is a property of the probe, not of any language model, while the training-checkpoint crossing is the model. The instrument is validated by reproduction first: on the Domany–Kinzel automaton the common-random-numbers damage field matches an independent prediction bit-exactly.","pith_inferences":["If the exact-zero null is backend-dependent, every damage-spreading reading inherits that dependence: moving to approximate or nondeterministic sampling would require re-validating all conclusions, not just the null.","The same two-way variation test could be applied to existing self-feeding practices such as self-consistency, iterated refinement, and agentic loops, deciding which of their readouts are construction artifacts.","The bit-exact Domany–Kinzel reproduction suggests a general template for model-dynamics instruments: calibrate on a system with a known answer before reading anything off a language model.","One testable extrapolation is that the diversity–$\\lambda_{\\mathrm{ca}}$ relation, which the paper finds holds across temperatures and checkpoints, might serve as a cheap proxy for training progress in model families without public checkpoints; that is an editorial extension, not the paper's claim."],"forward_implications":["Every reading taken from a self-feeding loop, such as agreement rates, convergence speeds, or self-correction success, must be classified by the variation test before it is attributed to the model.","A sharp, reproducible, precisely measured transition in an iterated probe can be entirely a property of the probe; precision and reproducibility do not imply that the model is the source.","$\\lambda_{\\mathrm{ca}}$ is a developmental quantity: within one model's training it crosses zero reproducibly, but across models it does not rank them consistently.","The attractor share is the model-determined reading that survives construction variation, ranking models consistently across lattice constructions.","Estimator defects of a recognizable class, including saturated predictors, noise denominators, and undefined values treated as zero, can produce confident wrong numbers, so estimators must be gated at their own geometry."],"supporting_citations":[{"why":"Supplies the Glauber-dynamics-on-token-sequences substrate the paper re-couples; the contrast is between maximal coupling there and common random numbers here.","marker":"[Sana et al., 2026]"},{"why":"Provides the Domany–Kinzel identity used for the bit-exact calibration of the damage field.","marker":"[Domany and Kinzel, 1984]"},{"why":"Source of damage spreading and Lyapunov exponents in cellular automata, the inherited machinery the instrument adapts.","marker":"[Bagnoli et al., 1992]"},{"why":"Self-consistency, the motivating self-feeding probe whose readings the discriminator targets.","marker":"[Wang et al., 2022]"},{"why":"Iterated refinement, the motivating self-feeding probe whose readings the discriminator targets.","marker":"[Madaan et al., 2023]"},{"why":"Supplies the light-cone analogue the paper uses to frame the kinematic bound on damage propagation.","marker":"[Lieb and Robinson, 1972]"}],"fun_headline_variants":["Self-feeding LLM probes mix model and artifact: one test separates them","Probe-only phase transition exposed by vary-one-hold-one test","One test tells probe artifacts from model signals in self-feeding LLMs","Vary one factor, hold the other: separates probe from model","Three-decimal phase transition is probe, not model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The exact-zero null, that two rings with identical windows handed the same random numbers draw exactly the same token, is load-bearing; if a backend has any nondeterminism or approximate sampling, undamaged twins will diverge and differences can no longer be attributed solely to the injected flip.","fun_headline_variants_meta":{"raw":{"variants":["Self-feeding LLM probes mix model and artifact: one test separates them","Probe-only phase transition exposed by vary-one-hold-one test","One test tells probe artifacts from model signals in self-feeding LLMs","Vary one factor, hold the other: separates probe from model","Three-decimal phase transition is probe, not model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001475,"raw_usage":{"total_tokens":6036,"prompt_tokens":1163,"completion_tokens":4873,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":779,"completion_tokens_details":{"reasoning_tokens":4782}},"tokens_in":779,"tokens_out":4873,"duration_ms":34185,"temperature":1.0,"reasoning_tokens":4782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:41:06.401038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Advance two undamaged rings under common random numbers on a backend with nondeterministic or approximate sampling; if their divergence is ever nonzero, the exact-zero null fails and all damage-spreading readings are contaminated.","supporting_citations":[{"cited_title":"Mixing times of glauber dynamics on masked language models, 2026","cited_arxiv_id":null,"evidence_quote":"Supplies the Glauber-dynamics-on-token-sequences substrate the paper re-couples; the contrast is between maximal coupling there and common random numbers here."},{"cited_title":"Damage spreading and lyapunov exponents in cellular automata","cited_arxiv_id":null,"evidence_quote":"Source of damage spreading and Lyapunov exponents in cellular automata, the inherited machinery the instrument adapts."}],"review_version":1}