{"id":"936b9e08-c4f3-470b-beff-79390f365ac0","arxiv_id":"2608.10288","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"full","parameter_count":2,"one_line_summary":"After training, a 110M-parameter power-law attention model's learned scoring operator becomes nearly input-invariant, so inference can cache it; the paper proves this collapse conditionally, measures it at 1e-6 and below, and machine-checks selected proofs in Lean 4.","lead":"This paper analyzes a power-law attention mechanism that learns a custom score operator from every input, and proves it contains standard attention as a special case. After training, the operator becomes nearly constant, so it can be cached at inference; the result is backed by proofs, a Lean 4 formalization, and measurements on a released model.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The collapse claim rests on invariance measured on one checkpoint under greedy decoding of four short prompts; that self-generated tube is too narrow to establish collapse as a general inference property.","rationale":"I read the paper in good faith. The conditional collapse theorem and the sufficiency direction are sound; the Lean formalization covers the algebraic cores; and the paper's epistemic labeling is unusually careful, with Appendix D genuinely measuring cache fidelity on the audited checkpoint. My concern is not internal inconsistency but the scope of the empirical premise that activates the theorem. The reader's weakest_assumption identifies the same empirical-coverage issue, and I agree with it. The three-stage contraction hypothesis is also sampled rather than certified, but it is explanatory rather than load-bearing for the collapse claim itself: even if the mechanism were wrong, the measured bitwise GLM invariance would still support the conditional conclusion on the tested inputs. I do not think this requires changing the CONDITIONAL verdict, because the concern is about generalization beyond the tested tube rather than correctness of what is claimed within its scope. The proposed stochastic, multi-prompt, long-context audit would settle whether the empirical collapse extends to the broader inference distribution the title implies.","tokens_in":54249,"tokens_out":5428,"duration_ms":65002,"concrete_test":"Run the Appendix D.7 cached-versus-recomputed audit under stochastic sampling on a held-out prompt battery: 50 prompts spanning varied domains, registers, and lengths from 32 to 1024 tokens, sampling 200 tokens per prompt at temperature 1 with full recomputation of GLM at every step. Record per-step relative GLM deviation, per-step maximum logit deviation against the minimum realized top-two margin, and token-level agreement between cached and uncached decoding. If any sampled token differs, or any logit deviation exceeds half the realized margin on any step, the collapse is not a general inference property; if all runs stay bitwise or safely below margins, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that at inference the deep PLGA subnetwork can be replaced by a cached tensor operator because the deductive outputs are input-invariant. The conditional theorem (Theorem 5.4(ii)) is sound, but the measurement that activates it is narrow. Appendix D.7 compares cached versus recomputed GLM only for greedy decoding of 48 tokens on 4 prompts, with per-layer bitwise equality on 960 comparisons; the order-parameter measurements of D.9 use 5 prompts with stochastic continuations but do not compare cached versus recomputed sampled decisions. The greedy tube is unusually narrow: the same checkpoint's greedy continuations have duplicate 4-gram fractions of 0.49-0.80, so later contexts are dominated by repeated tokens. If invariance is a property of the visited tube rather than of the trained map, this tube is not representative of general inference. The contrast checkpoint (110M-1) shows 74.5% historical-row movement, so the architecture does not guarantee collapse; it is an empirical property of one training run. The contraction mechanism of Section 5.4 is not the load-bearing point: even if the proposed origin is incomplete, the measured collapse would stand; what remains exposed is the generality of the empirical premise that exact input invariance holds across the deployed distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an analytical treatment of Power Law Graph Attention (PLGA) and the PLDR-LLM architecture. It proves several structural properties: PLGA contains scaled dot-product attention exactly at the identity operator (Theorem 5.4(i)); the generator A is strictly entrywise positive with Perron-Frobenius structure (Theorem 3.8); a rank-one singularity condition holds exactly (Proposition 3.9); the DAG regularizer has the NOTEARS walk-counting form with an explicit positivity obstruction (Theorem 4.4, Remark 4.5); and a commutant characterization for rotary embeddings identifies which operators preserve relative-position dependence (Proposition 4.1). The central result is Theorem 5.4(ii): if every deductive output is exactly input-invariant, then the inference map coincides with a generalized SDPA with constant learned bilinear forms. The empirical companion is a numerical audit on a released 110M checkpoint (Appendix D), reporting bitwise invariance of the operator under greedy decoding and agreement between cached and recomputed inference, alongside a contrast checkpoint showing historical-row movement. The paper also proposes a conditional three-stage mechanism for the invariance (rotary twirl, concentration, row-map contraction) and frames self-organized criticality as a phenomenological hypothesis. Claims are explicitly labeled as theorems, conditional theorems, measurements, or conjectures, with selected proof cores machine-checked in Lean 4.","tokens_in":54493,"tokens_out":7916,"duration_ms":79035,"significance":"The paper is exceptionally careful in labeling the epistemic status of each claim, and it provides an unusually detailed reproducibility package: pinned checkpoints and commits, Lean 4 formalization of elementary proof cores, and a numerical audit with raw arrays and hashes. The exact containment of SDPA in the PLGA family and the conditional inference-collapse theorem are clean conceptual contributions that would be of interest to the community working on attention variants and efficient inference. The empirical audit is honest and self-critical, explicitly reporting where bounds fail to close margins and which mechanisms are not certified. The main limitation is the narrowness of the empirical basis for the title-level 'empirical collapse at inference' claim: a single checkpoint, a small number of prompts, and greedy decoding only. If the title claim is narrowed accordingly, the paper's contribution is solid and publishable.","major_comments":[{"comment":"The title's unqualified claim 'empirical collapse at inference' is stronger than the evidence reported in the manuscript. Exact invariance is measured on a single released 110M checkpoint (PLDR-LLM-v51-SOC-110M-5) under greedy decoding of 48 tokens on four prompts, with 960 per-layer bitwise comparisons of the recomputed versus cached GLM (Appendix D.7). The stochastic-continuation order-parameter measurements in Appendix D.9 do not compare cached versus recomputed sampled decisions. Appendix D.10 shows that a contrast checkpoint of the same architecture (PLDR-LLM-v51-SOC-110M-1) exhibits 74.5% historical-row movement, demonstrating that collapse is not an architectural identity. Because the measured invariance is the empirical premise that activates Theorem 5.4(ii), this narrow tube is a load-bearing limitation of the central advertised claim. I recommend either explicitly restricting the title and abstract to 'empirical collapse at inference on the released checkpoint under greedy decoding' or substantially widening the audit to include more checkpoints, longer and more diverse prompts, and cached-versus-recomputed comparisons under stochastic sampling.","section":"Title/Abstract; Appendix D.7, D.9, D.10"}],"minor_comments":[{"comment":"The symbol Ψ is used in equation (4.15) and in the surrounding text ('the deductive chain Ψ') without being defined; please define it explicitly, ideally before its first use in Remark 4.9.","section":"Section 4.5, Remark 4.9"},{"comment":"The genealogical paragraph comparing PLGA to PaTH attention is extremely long and difficult to parse; a structured summary or table would improve readability without losing the careful distinctions being drawn.","section":"Section 1, Related Work"},{"comment":"The audit reports the margin check as '2B = 7.2e-5 < 6.0e-3', but Corollary 5.13 gives both a 2B criterion and a √2B criterion depending on whether the bound is in ℓ∞ or ℓ2; please state explicitly which norm the measured B refers to and which criterion is being applied.","section":"Appendix D.7 and Corollary 5.13"},{"comment":"The phrase 'the assembled proxy misses the decoding margin' is cryptic without reading Section 5.4 and Appendix D.7; adding a brief explanatory clause (for example, that the worst-case bound is far too large to certify bit-identical decoding, while measured deviations are far below the margin) would make the abstract more accessible.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript comes from a single-author research program and presents the author's own architecture. The verification infrastructure is unusually strong: pinned repos, Lean 4 proof cores, and a detailed audit with raw data. The main obstacle to acceptance is the mismatch between the unqualified title claim and the narrow empirical basis for the collapse phenomenon. If the authors are willing to narrow the claim in the title and abstract, the paper would be acceptable; if they prefer to keep the broad claim, they need a substantially wider empirical study. I also note that the related-work section is extensive and cites the closest prior art fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nRead the PLGA paper. The formal work is genuine, the audit is a model of how to report checkpoint-level measurements, and the paper's own claims are more carefully scoped than its title. The reader's CONDITIONAL verdict is right; the stress-test worry about narrow coverage is fair as far as it goes, but it doesn't break the paper on its own terms, because the invariance is explicitly a measured hypothesis, not a theorem.\n\nWhat's actually new: the commutant characterization of relative-position invariance under RoPE (Prop. 4.1) is clean, with a correctly stated nonresonance hypothesis. The inference-collapse theorem (Thm. 5.4(ii)) is honestly sufficiency-only, and the SDPA-at-G=I containment is exactly what it claims. The epsilon-LayerNorm analysis (Lem. 5.11) is careful, and measuring the scale error directly rather than trusting the proxy is a nice touch. Selected proof cores are machine-checked in Lean with an axiom audit, and the Appendix C coverage table is unusually explicit about what is and isn't kernel-checked.\n\nAppendix D earns real credit. Singular-value spectra instead of float-zero determinants, trained-versus-initialized controls, bitwise cached-versus-recomputed comparison on a pinned checkpoint, and — most refreshingly — an explicit statement that its own perturbation budget fails by ~45 orders of magnitude to certify decoding. That is honest quantification of a mechanism, not a disguised certificate.\n\nSoft spots, in proportion: the empirical base is one checkpoint, eight prompts, four of them for greedy fidelity with heavily repetitive continuations (duplicate 4-gram fractions 0.49–0.80). The contrast checkpoint showing 74.5% historical-row movement confirms collapse is a property of one training run, not the architecture. The Stage-3 contraction mechanism is sampled, not certified — but it's not load-bearing; the measured collapse stands even if the proposed origin is wrong. The SOC phase labeling is acknowledged circular, which mitigates but doesn't erase the title's framing debt.\n\nWho gets value: attention-theory people for Prop. 4.1 and Thm. 5.4; anyone running model audits can learn from Appendix D; the epistemic-labeling scheme is worth stealing.\n\nRecommendation: send it out. The right referee will push on generality of the empirical premise and on what's consolidation versus new, but the formal core is sound and the empirical reporting is a cut above the usual standard.","headline":"A carefully scoped theory-plus-audit paper: the formal results are clean and partly Lean-checked, the collapse is honestly measured on one checkpoint, and the real gap is between the title's generality and a narrow empirical base.","tokens_in":55030,"tokens_out":3660,"would_cite":true,"duration_ms":36829,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","15B48","05C50","82C27"],"pacs":[],"model":"deepseek-v4-flash","headline":"Attention built on a learned, input-generated metric operator contains standard attention exactly, and at inference collapses to a cached tensor.","keywords":["power law graph attention","scaled dot-product attention","learned bilinear operator","input invariance","inference collapse","rotary position embeddings","commutant","self-organized criticality"],"falsifier":"Run the same Appendix D audit on a released near-critical checkpoint at a different scale (e.g. the 41B-token 110M model with a different prompt distribution, or any larger released variant): if the recomputed G_LM differs from the cached value by more than the decoding margin on any step, or if the composite Jacobian of the row map phi exceeds 1 at any visited row, then the collapse is an artifact of the specific audited checkpoint rather than a trained property of the family.","tokens_in":54018,"feed_emoji":"🧮","tokens_out":2235,"duration_ms":22608,"temperature":0.7,"pith_summary":"The paper gives a full analytical account of Power Law Graph Attention (PLGA), the attention mechanism inside the PLDR-LLM language model family, in which the fixed bilinear form of scaled dot-product attention (SDPA) is replaced by a learned operator generated from the input itself. The central theoretical claim is that SDPA is exactly the identity-operator point of this larger family, and that if a trained model's learned operators are exactly input-invariant, then the entire deep nonlinear subnetwork that produces them can be deleted at inference and replaced by a single cached constant tensor with no change to the model's outputs. The paper further reports that on a released checkpoint this invariance is measured at floating-point resolution, so the cached model is empirically indistinguishable from the full recomputed model on the tested workloads. The broader ambition is a family of models in which the attention geometry itself is learned, regularized, measured, and cached, rather than fixed a priori.","feed_headline":"Learned attention operator collapses to a cached tensor at inference","feed_subtitle":"When trained invariant, the deep metric subnetwork provably reduces to a fixed constant operator; measured bitwise on a released checkpoint.","key_machinery":"The central object is the learned bilinear score operator G_LM, a d_k-by-d_k matrix per head produced through a chain: the rotated query Gram D_Q feeds a deep row-wise residual network (the metric learner), whose output generator A is transformed by an elementwise power law with learned exponents and a learned affine superposition into G_LM; attention scores are then q^T G_LM k / $\\sqrt$(d_k). A second load-bearing object is the shared row map $\\varphi$ of the metric learner: because phi acts identically on every row and every head of a layer, its collapse to a locally constant map forces the generator A to be rank-one and identical across heads, which is exactly the condition measured on the released checkpoint. The proof of collapse rests on the exact input-invariance hypothesis, with propagation of small perturbation errors bounded by explicit constants.","core_discovery":"The central discovery is an inference-collapse theorem: if every deductive output of PLGA is exactly input-invariant, taking values G*(l,i), then the inference map of the full architecture coincides with the map of a generalized SDPA in which the learned subnetwork (3.3)-(3.6) is replaced by the constant operators G*(l,i). At the identity operator G = I this recovers ordinary scaled dot-product attention exactly. Empirically, on the released near-critical checkpoint, the recomputed G_LM equals the cached value bitwise at every compared decoding step, the greedy token choices agree at every step, and the logit deviations lie roughly two orders of magnitude below the smallest realized decision margin. The same architecture at a different checkpoint shows substantial historical-row movement (74.5% of compared entries), so the collapse is a trained property, not an architectural identity.","pith_inferences":["A plausible testable extension: on checkpoints at intermediate training steps, the measured historical-row movement and the order parameter should decay together with the composite Jacobian of the metric learner's row map, which would locate the collapse along the training trajectory rather than only at the end state.","The commutant residual measured in Appendix D suggests a causal experiment not performed in the paper: project a trained G* onto the RoPE commutant and measure the behavioral change in decoding; if the off-commutant energy is largely unused, the measured absolute-position sensitivity is occupancy without function, which would refine the significance of Corollary 7.1.","If invariance is a trained attractor property rather than an architectural identity, then the same collapse phenomenon, or its absence, should be reproducible in non-SOC training regimes at larger scale, which would determine whether the criticality framework of Section 6 is essential to the effect or merely correlated with it."],"forward_implications":["If the invariance theorem holds for a trained model, the entire PLGA subnetwork can be excised at inference, replaced by cached operators G*(l,i), without changing any output; this is what makes the G-cache exact relative to the prompt-frozen KV-cache, and empirically bitwise faithful to full recomputation on the audited checkpoint.","SDPA is not a rival architecture but the identity-operator point of the PLGA family, so any comparative statement between the two is a statement about a family and its base point, not about separate designs.","Under nonresonance of the standard rotary frequencies, a constant operator G* leaves scores dependent only on relative position exactly when G* lies in the rotation commutant; for a generic learned G* the trained operator carries absolute-position information, with d_k^2 - d_k off-commutant dimensions per head beyond what a pre-RoPE-linearly-reparameterized SDPA head can express.","The blockwise training objective used by the released pipelines contains target-exposed auxiliary summands for historical rows; the paper shows they are not next-token log probabilities as a protocol unless historical-row prefix consistency holds, which the global Gram does not supply in general and which is measured to fail on a released checkpoint.","The three-stage mechanism (rotary twirl, concentration, row-map contraction) provides a quantitative, ingredient-tagged explanation of how input invariance can arise during training, with each stage's measurable diagnostics computed directly on the released checkpoint rather than assumed."],"supporting_citations":[{"why":"Defines scaled dot-product attention, the base point of the PLGA family and the target of the exact-containment claim in Theorem 5.4(i).","marker":"[43]"},{"why":"Supplies the rotary position embedding structure whose commutant is characterized in Proposition 4.1 and used in the collapse and codimension arguments.","marker":"[38]"},{"why":"Provides the NOTEARS walk-counting characterization used in Theorem 4.4 for the DAG regularizer.","marker":"[50]"},{"why":"Supplies the Perron-Frobenius theorem used in Theorem 3.8 and the Weyl singular-value inequality used in Proposition 3.9.","marker":"[18]"},{"why":"Supplies the matrix Bernstein inequality used in the Stage-2 concentration bound of Proposition 5.10.","marker":"[41]"},{"why":"Source of the released checkpoint, the cached-versus-uncached benchmark scores, and the operator-invariance empirical observations on which the collapse claims rest.","marker":"[13]"},{"why":"Source of the order-parameter definition, the near-critical/sub-critical phase separation, and the m ~ 10^-6 to 0 measurements cited as empirical support for the collapse.","marker":"[14]"},{"why":"Supplies the 1-Lipschitz property of softmax used in the perturbation bound of Proposition 5.7.","marker":"[9]"}],"fun_headline_variants":["Trained invariance collapses attention to a fixed operator","Inference collapse: recomputed attention matches cached bitwise","PLGA becomes SDPA with constant operator when invariant","Exact collapse theorem: attention reduces to a cached tensor","Invariance provably reduces attention to a constant operator"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing empirical premise is that the invariance measured in Appendix D on one released 110M checkpoint with eight fixed prompts is representative of PLDR-LLM inference more broadly; the causal mechanism additionally assumes, as an unproved hypothesis, that the trained row map $\\varphi$ is contractive on the visited tube, so if either the measured invariance fails to generalize or the contraction hypothesis is false on a different checkpoint, the title-level claim of empirical collapse loses its support.","fun_headline_variants_meta":{"raw":{"variants":["Trained invariance collapses attention to a fixed operator","Inference collapse: recomputed attention matches cached bitwise","PLGA becomes SDPA with constant operator when invariant","Exact collapse theorem: attention reduces to a cached tensor","Invariance provably reduces attention to a constant operator"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000524,"raw_usage":{"total_tokens":2593,"prompt_tokens":1068,"completion_tokens":1525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":1447}},"tokens_in":684,"tokens_out":1525,"duration_ms":10670,"temperature":1.0,"reasoning_tokens":1447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:27.696185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same Appendix D audit on a released near-critical checkpoint at a different scale (e.g. the 41B-token 110M model with a different prompt distribution, or any larger released variant): if the recomputed G_LM differs from the cached value by more than the decoding margin on any step, or if the composite Jacobian of the row map phi exceeds 1 at any visited row, then the collapse is an artifact of the specific audited checkpoint rather than a trained property of the family.","supporting_citations":[{"cited_title":"Gomez, Łukasz Kaiser, and Illia Polosukhin,Attention is all you need, Advances in Neural Information Processing Systems 30 (NIPS), 2017, pp","cited_arxiv_id":null,"evidence_quote":"Defines scaled dot-product attention, the base point of the PLGA family and the target of the exact-containment claim in Theorem 5.4(i)."},{"cited_title":"Xing,DAGs with NO TEARS: Continuous optimization for structure learning, Advances in Neural Information Processing Systems 31 (NeurIPS), 2018, pp","cited_arxiv_id":null,"evidence_quote":"Provides the NOTEARS walk-counting characterization used in Theorem 4.4 for the DAG regularizer."},{"cited_title":"Horn and Charles R","cited_arxiv_id":null,"evidence_quote":"Supplies the Perron-Frobenius theorem used in Theorem 3.8 and the Weyl singular-value inequality used in Proposition 3.9."},{"cited_title":"Tropp,An introduction to matrix concentration inequalities, Foundations and Trends in Machine Learning8(2015), no","cited_arxiv_id":null,"evidence_quote":"Supplies the matrix Bernstein inequality used in the Stage-2 concentration bound of Proposition 5.10."},{"cited_title":"PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inference","cited_arxiv_id":"2502.13502","evidence_quote":"Source of the released checkpoint, the cached-versus-uncached benchmark scores, and the operator-invariance empirical observations on which the collapse claims rest."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the order-parameter definition, the near-critical/sub-critical phase separation, and the m ~ 10^-6 to 0 measurements cited as empirical support for the collapse."}],"review_version":1}