{"id":"15a7fb77-f9bb-41c1-9193-02ee28490579","arxiv_id":"2508.06776","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A claim that null-space leakage yields provable LLM drift detection is undermined by an inverted eigenvalue inference and a sketched regret proof, with no empirical validation.","lead":"This theory-only paper proposes detecting LLM drift by watching energy that appears in the null spaces of layer activations, without labels or outputs. It proves a set of bounds and thresholds, but its main theorem contains a false inference and there are no experiments, so the central guarantee fails as stated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's Variance-Leak inference is reversed: NVL≥ε implies λmax≥ε/kℓ, not λmin. A 2×2 counterexample falsifies the claimed certification.","rationale":"The paper's central claim is that null-space leakage yields concrete, testable guarantees on representational change, with Theorem 1 as the foundation. The load-bearing condition for that claim is that a positive NVL forces a positive smallest eigenvalue of the perturbation Gram matrix. Theorem 1's proof establishes the correct sandwich inequality, but the 'in particular' inference is reversed: it concludes λmin ≥ ε/kℓ from NVL ≥ ε, whereas the inequality kℓ λmin ≤ NVL cannot support that conclusion. A simple 2×2 counterexample shows NVL can be arbitrarily positive while λmin(G)=0, because NVL measures projection onto a subspace, not the smallest singular value of the full perturbation. This is an internal inconsistency, not a disagreement with external consensus, and it directly invalidates the framework's headline guarantee. The reader's verdict of REJECT is therefore correct. However, the reader's designated weakest_assumption was the Gaussian null model of §4.3, which is a different (and secondary) issue; the reader's rationale did mention the reversed inference, so there is partial alignment. Since our concern only strengthens the existing REJECT verdict, the recommended verdict is UNCHANGED. No independent verification (e.g., machine-checked proofs or experiments) exists to offset this error. We credit the paper for clearly stating assumptions and for some correct auxiliary results (e.g., Theorem 5's rank-leak bound appears valid), but these do not rescue the central Variance-Leak certification. The concrete test above is decisive and easy to reproduce.","tokens_in":12480,"tokens_out":4778,"duration_ms":54694,"concrete_test":"Run the explicit 2×2 instance: H=[[1,0],[0,0]], V0=[0,1]ᵀ, ΔH=[[0,1],[0,0]]. Compute NVL = ||ΔH V0||_F² = 1, G = ΔHᵀΔH = [[0,0],[0,1]], so λmin(G)=0. Set ε=0.5 and kℓ=1. The theorem's claimed implication 'NVL≥ε ⇒ λmin≥ε/kℓ' would require 0 ≥ 0.5, a contradiction. More generally, re-examine the proof step: from kℓ λmin ≤ NVL and NVL ≥ ε, no lower bound on λmin follows; the correct consequence is λmax ≥ ε/kℓ. This single counterexample falsifies Theorem 1 as stated, so the central certification claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 Theorem 1 proves the correct sandwich kℓ λmin(G) ≤ NVLℓ ≤ kℓ λmax(G), but then states: 'if NVLℓ≥ε then λmin(G)≥ε/kℓ'. This inference is logically reversed. The left inequality only gives kℓ λmin(G) ≤ NVLℓ, so a large NVL is compatible with λmin(G) = 0; the conclusion one can draw is λmax(G) ≥ ε/kℓ. The paper's own 'Equivalently, any nonzero NVL implies a strictly positive smallest eigenvalue' is also false. Counterexample: let H = diag(1,0), so V0 = e2 and kℓ=1; take ΔH = [[0,1],[0,0]]. Then NVL = ||ΔH e2||² = 1, while G = ΔHᵀΔH = diag(0,1), so λmin(G) = 0. With ε=0.5, NVL≥ε but λmin(G)≥ε/kℓ fails. This is not a modeling assumption or a matter of consensus; it is an internal mathematical error in the central theorem. The abstract and discussion rely on Theorem 1 as the framework's 'concrete, testable guarantee', so the central claim collapses as stated. The Gaussian-null thresholds in §4.3 and the regret analysis are separate, but they cannot repair the reversal. The reader's weakest_assumption about the Gaussian null is a legitimate secondary concern, but the load-bearing issue is the reversed implication.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Zero-Direction Probing (ZDP), a theory-only framework for detecting drift in LLM representations by monitoring energy that appears in the null space of base-model activation matrices. It states assumptions A1–A6, defines probe functionals NVL/SNL/FNC/BINA, and claims four main results: the Variance–Leak Theorem (Thm 1), Fisher Null-Conservation (Thm 3), a Rank–Leak bound for low-rank updates (Thm 5), and a logarithmic-regret guarantee for online null-space trackers (Thm 4). It also derives random-matrix thresholds under a Gaussian null model. The central advertised guarantee is that if the null-variance leak NVL_ℓ is at least ε, then the smallest eigenvalue of the perturbation Gram matrix G is at least ε/k_ℓ, which the paper presents as a concrete, calibration-free certification of drift.","tokens_in":12898,"tokens_out":10073,"duration_ms":108000,"significance":"The idea of extracting signal from zero-variance directions is conceptually appealing, and some isolated pieces—notably the Rank–Leak bound via principal angles (Thm 5) and the rotational-invariance calculation (Prop. 7)—are correct as stated. However, the paper's main load-bearing claim, the Variance–Leak Theorem, is invalid: the proof establishes only a sandwich inequality from which the stated implication does not follow. A concrete counterexample shows that large NVL can coexist with λ_min(G)=0, so the advertised certification guarantee collapses. Since the abstract, contributions, and discussion all rest on this theorem, the manuscript in its present form is not sound. The Gaussian null thresholds and the online regret analysis are separate contributions, but they cannot repair the reversed central implication.","major_comments":[{"comment":"The implication 'if NVL_ℓ ≥ ε then λ_min(G) ≥ ε/k_ℓ' is logically reversed. The proved sandwich k_ℓ λ_min(G) ≤ NVL_ℓ ≤ k_ℓ λ_max(G) only permits the conclusion λ_max(G) ≥ ε/k_ℓ from a lower bound on NVL. The subsequent claim that any nonzero NVL implies a strictly positive smallest eigenvalue is false: take H = diag(1,0), V_0 = e_2, k_ℓ = 1, ΔH = [[0,1],[0,0]]. Then NVL = 1 but G = ΔHᵀΔH = diag(0,1), so λ_min(G)=0. This invalidates the paper's central certification claim and the abstract's 'concrete, testable guarantees on representational change.'","section":"§4.1, Theorem 1 and Eq. (1)"},{"comment":"The contributions list states that Fisher Null-Conservation (Thm 3) proves 'the second-order KL contribution arises only from components outside the base image space.' The theorem itself proves the opposite: the second-order KL contribution arises only from the component of Δθ lying in im(H_ℓ), and perturbations confined to ker(H_ℓ) are second-order KL-silent. This is not a typo-level discrepancy; it reverses the content of a headline theorem and must be corrected.","section":"§1 Contributions, item 2"},{"comment":"The displayed Laurent–Massart bound appears to have the wrong scaling. With m = nk degrees of freedom, the tail bound is 2√(mx)/n = 2√(kx/n), not 2√(kx)/n as typeset. The same scaling error appears in the denominator lower bound and propagates into inequalities (2) and (3). As written, the thresholds are too small and the claimed false-positive control is not achieved. Please correct the algebra or clarify the notation.","section":"§4.3, Lemma 2 and Corollary 1"},{"comment":"The main convergence estimate, Eq. (4), is asserted via 'standard analysis' and an ODE method without a complete proof. Since Theorem 4 is presented as one of the paper's principal contributions and the manuscript is deliberately theory-only, this is a substantial gap. Either supply the full argument for the Oja-type subspace contraction or downgrade the statement to a conjecture with supporting heuristics.","section":"§4.5, Theorem 4"}],"minor_comments":[{"comment":"The abstract says results hold 'Under assumptions A1–A6,' but A4–A6 are introduced only in §4.4 and are not needed for Theorems 1, 3, or 5. Please state which theorems require which assumptions.","section":"Abstract / §4.0"},{"comment":"References [15] and [18] both refer to 'LoRA-Null' but list different author teams; please reconcile. Several other references appear to have placeholder-style author lists; please verify all entries.","section":"References"},{"comment":"There are numerous typographical issues: 'We prove' capitalized mid-sentence, 'theVariance' missing space, 'D ℓ' introduced without clear definition in §3.1 (though it is used later), and inconsistent use of n vs. m for batch size in the online sections. A careful copyedit is needed.","section":"Throughout"},{"comment":"The claim of 'calibration-free thresholds' is too strong. Corollary 1 depends on the Gaussian null model and on the variance σ² (even if σ²=1 is chosen for illustration). The paper should state clearly that these are thresholds under a modeling assumption, not universally calibration-free alarms.","section":"§4.3"}],"recommendation":"reject","confidential_remarks":"The central theorem is false as stated, and a corrected version would not deliver the advertised certification. The remaining results are not enough to sustain the paper's main claims. I would not recommend acceptance or minor revision; the authors would need to substantially reframe the contribution and reconsider what can actually be guaranteed from NVL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Name],\n\nThe headline: this paper has a genuinely useful framing for drift detection, but its central theorem is wrong, and the mistake is not a technicality.\n\nWhat's actually new: the idea of monitoring null spaces of activations as drift witnesses, bundling NVL/SNL with RMT thresholds, and the Rank-Leak bound (Thm 5) are reasonable. Theorem 5 is correct and clean, using principal angles. Lemma 2's chi-square tail is standard but correctly applied. The paper is honest that it is theory-only and flags its own limitations.\n\nThe fatal problem: Theorem 1 proves k λmin(G) ≤ NVL ≤ k λmax(G) but then claims NVL ≥ ε implies λmin(G) ≥ ε/k. That is the reversed inequality. A 2x2 counterexample (H=diag(1,0), ΔH=[[0,1],[0,0]]) gives NVL=1, λmin(G)=0. So the 'certification' claim collapses. The abstract also reverses the Fisher Null-Conservation result: it says KL contributions come from 'outside' the image space, while the theorem says they come from 'in' it. That misstatement matters for readers.\n\nSecondary concerns: the Gaussian null for thresholds (Lemma 2) is plausible but unverified for transformer activations; the paper admits non-Gaussian extension is future work. The online regret proof (Thm 4) is a sketch: it invokes 'standard analysis' for contraction and has a measurability issue because the tracker's projector Pt depends on the current batch Ht, so conditioning on the past filtration doesn't give tr((Pt-P*)Σ). There is no empirical validation, but the author says that is intentional.\n\nNet: the package is interesting as an idea, but the load-bearing result is false as stated. A serious referee should look at it, mostly to catch this class of error. Not citable in its current form.","headline":"A promising drift-detection framework whose central theorem reverses its own inequality; the main claim fails, but some pieces are salvageable.","tokens_in":13315,"tokens_out":2630,"would_cite":false,"duration_ms":28552,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["15A18","60B20","62H25","68T07","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that monitoring the zero-variance directions of transformer activations yields provable, label-free bounds on representational drift.","keywords":["null-space probing","model drift","large language models","spectral null-leakage","Fisher information","random matrix theory","low-rank adaptation","online subspace tracking"],"falsifier":"Take a real transformer layer, compute an exact or high-accuracy null basis $V_{0,\\ell}$ from base activations, apply a small fine-tune, and compare the empirical distribution of $\\|\\hat H_\\ell V_{0,\\ell}\\|_F^2$ with the chi-square/MP tail predicted under the Gaussian null with the same $(n,d,k)$. If the empirical false-alarm rate systematically exceeds the predicted rate, the calibration-free claim is falsified; conversely, matching tails would support the framework.","tokens_in":12383,"feed_emoji":"📡","tokens_out":17436,"duration_ms":161854,"temperature":0.7,"pith_summary":"This paper tries to establish that the null spaces of transformer layer activations—the directions where the base model carries zero variance—are clear witnesses of representational drift. It proves a chain of theorems: any energy that appears in those silent directions lower-bounds the perturbation's Gram matrix spectrum (the Variance–Leak Theorem); second-order KL divergence is blind to perturbations confined to the null space (Fisher Null-Conservation); and low-rank updates leak into silent directions exactly when their B-factor is not orthogonal to the null space (Rank–Leak Bound). It also derives parameter-free alarm thresholds for a Spectral Null-Leakage metric under a Gaussian null model, plus logarithmic-regret guarantees for online null-space trackers. The manuscript is deliberately theory-only, so the value is in the proof chain: if correct, it offers a label-free, output-free route to certifying when a fine-tuned model has internally changed.","feed_headline":"Null-space energy certifies LLM drift without labels","feed_subtitle":"Fine-tuned models leak into base activations' zero-variance directions; this paper proves those leaks bound the change.","key_machinery":"The central object is the right-null basis $V_{0,\\ell} = \\ker(H_\\ell)$ of a transformer layer's activation matrix, with the quadratic functional $NVL_\\ell = \\|\\hat H_\\ell V_{0,\\ell}\\|_F^2$. The identity $\\hat H_\\ell V_{0,\\ell} = \\Delta H_\\ell V_{0,\\ell}$ reduces null-space energy to a trace of the perturbation Gram matrix $G = \\Delta H_\\ell^\\top \\Delta H_\\ell$, putting Rayleigh–Ritz eigenvalue bounds in play. The Fisher analysis uses the restricted Fisher matrix $F_\\parallel = P_\\parallel^\\top F(h) P_\\parallel$ to show that only the image-space component of $\\Delta\\theta$ contributes to second-order KL. The low-rank analysis uses an SVD of the LoRA B-factor to convert leakage into $\\cos^2$ p","core_discovery":"The central claim is that 'listening to silence' works: compute the right-null basis $V_{0,\\ell}$ of the base activation matrix $H_\\ell$ at each layer, and measure how much perturbed activations $\\hat H_\\ell$ project onto that basis. The Variance–Leak Theorem states that $NVL_\\ell = \\|\\hat H_\\ell V_{0,\\ell}\\|_F^2$ satisfies $k_\\ell \\lambda_{\\min}(G) \\le NVL_\\ell \\le k_\\ell \\lambda_{\\max}(G)$ for $G = \\Delta H_\\ell^\\top \\Delta H_\\ell$, so a positive null-space energy is claimed to certify a strictly positive smallest eigenvalue of the perturbation Gram matrix. Fisher Null-Conservation shows the second-order KL term comes only from the component of the parameter perturbation lying in the image","pith_inferences":["Beyond the paper, the Variance–Leak inequality is a purely linear-algebraic statement about any pair of matrices $H$ and $H+\\Delta H$, so the certificate would transfer to CNNs, MLPs, or other architectures whose layer activations are matrices—the paper itself claims only transformers.","Beyond the paper, Fisher Null-Conservation suggests a constructive experiment: fine-tune a model with updates constrained to the base null space and measure downstream task behaviour; the theorem predicts second-order KL is unchanged, implying any task shift must come from higher-order terms.","Beyond the paper, SNL is a scale-free ratio and could be computed per layer to form a depth-resolved drift heatmap, showing not just that drift happened but where it first entered the network; the paper does not propose this diagnostic.","Beyond the paper, the Gaussian-null thresholds are directly testable: if real activations' projected energy has heavier tails than the chi-square prediction, the calibration-free claim would need to be replaced by a sub-Weibull or empirically estimated threshold, leaving the theorems intact but losing the parameter-free alarm."],"forward_implications":["According to the paper, any NVL alarm above a chosen threshold carries a certificate: the perturbation Gram matrix has smallest eigenvalue at least $\\epsilon/k_\\ell$, so the detector guarantees a nonzero spectral footprint rather than a heuristic score.","If Fisher Null-Conservation holds, a fine-tune whose parameter update stays in $\\ker(H_\\ell)$ is second-order KL-silent, so any measured KL increase must be accompanied by null-space leakage that NVL/SNL can detect.","If the Rank–Leak bound holds, LoRA-Null's zero-leakage initialisation is equivalent to $B^\\top V_{0,\\ell} = 0$, and the principal-angle formula quantifies how subsequent training rotates the update back toward silent directions.","If the Gaussian-null thresholds hold, SNL alarm levels are fixed in advance from $(n,d,k,\\alpha)$, enabling drift monitoring without historical ROC curves or label-based calibration.","If the online regret bound holds, streaming null-space trackers accumulate only $O(\\log T)$ excess leakage, so long-horizon monitoring does not drift out of statistical control."],"supporting_citations":[{"why":"Defines the LoRA-Null adaptation scheme; Theorem 5 formalises its zero-leakage condition and the paper's online null-aligned LoRA optimiser builds on it.","marker":"[18]"},{"why":"Supplies the random-matrix motivation and the small-singular-value perspective that the SNL baselines are designed to formalise.","marker":"[12]"},{"why":"Provides the free-probabilistic analysis of transformer activations used in Proposition 7 to compute the expected overlap between random subspaces.","marker":"[20]"},{"why":"The proof of Lemma 2 follows its Theorem 1.6 to obtain the Marchenko–Pastur edge tail for the projected Gaussian energy.","marker":"Benaych–Georges & Nadakuditi (2012, Thm 1.6)"}],"fun_headline_variants":["Proven: LLM drift leaks into null directions","Null-space probe certifies drift with zero labels","No labels needed: null-space math proves drift"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The a-priori drift thresholds (Lemma 2 and Corollary 1) assume the perturbed activation matrix has independent, identically distributed Gaussian entries, a null model that real transformer activations do not follow and that the paper does not test.","fun_headline_variants_meta":{"raw":{"variants":["Proven: LLM drift leaks into null directions","Null-space probe certifies drift with zero labels","No labels needed: null-space math proves drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1064,"prompt_tokens":712,"completion_tokens":352,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":304}},"tokens_in":456,"tokens_out":352,"duration_ms":4808,"temperature":1.0,"reasoning_tokens":304,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:32:15.829049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real transformer layer, compute an exact or high-accuracy null basis $V_{0,\\ell}$ from base activations, apply a small fine-tune, and compare the empirical distribution of $\\|\\hat H_\\ell V_{0,\\ell}\\|_F^2$ with the chi-square/MP tail predicted under the Gaussian null with the same $(n,d,k)$. If the empirical false-alarm rate systematically exceeds the predicted rate, the calibration-free claim is falsified; conversely, matching tails would support the framework.","supporting_citations":[{"cited_title":"Lora-null: Low-rank adaptation via null space for large language models","cited_arxiv_id":null,"evidence_quote":"Defines the LoRA-Null adaptation scheme; Theorem 5 formalises its zero-leakage condition and the paper's online null-aligned LoRA optimiser builds on it."},{"cited_title":"A Free Probabilistic Framework for Analyzing the Transformer-based Language Models","cited_arxiv_id":"2506.16550","evidence_quote":"Provides the free-probabilistic analysis of transformer activations used in Proposition 7 to compute the expected overlap between random subspaces."}],"review_version":1}