{"id":"3b6ff72d-c1f9-4761-8424-2886e9635cba","arxiv_id":"2502.00669","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Using a Markov-chain model of LLM fine-tuning, the paper claims a training-step bound that makes refusal states absorbing and proposes ensemble width as a substitute for alignment depth, but the proof is flawed.","lead":"Large language models are viewed as Markov chains, and the paper claims to prove that with enough fine-tuning, refusal states become absorbing, and that ensembles of shallow-aligned models can match deep alignment. The proof and toy experiments contain internal inconsistencies, and the step from the toy chain to real LLMs rests on unvalidated assumptions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.5 is internally inconsistent: with the paper's own T=65 example, Eq. (14) gives Q_T(r,r)≈1.51 and Q_T(r,s)≈−0.51, so the matrix is not stochastic and the claimed δ-absorbing bounds fail before the ad hoc normalization in Remark 4.6.","rationale":"I read the paper as trying to establish a theoretical guarantee that fine-tuning can make refusal states δ-absorbing, with the theorem then extended to cyclic augmentation and ensembles. For that central claim to hold, Theorem 4.5 must be valid at least for the toy Markov chain. It is not: the proof's cumulative sum is inconsistent with the stated update, and even granting the cumulative reading, the inequality in Eq. (16) does not follow from the hypotheses. The paper's own example numerically disproves the claimed bounds before any clipping is applied. This is an internal mathematical error, not a disagreement with field consensus. The reader's weakest_assumption pointed to Assumptions 3.1 and 3.4 bridging toy dynamics to real LLM fine-tuning; I partially agree that those assumptions are unvalidated, but I found an even more basic failure: the toy theorem itself is false, so the bridge is moot. Additional corroborating problems include Proposition 4.8's proof relying on the identity ΣP^tBP^{−t}=mB, which does not hold for general permutation matrices, and the toy experiment using α=γ=1, outside the theorem's 0<γ<1 hypothesis, while being presented as validation. The LLM experiments do not measure any Markov-chain quantity, and the deep-baseline selection is post hoc, but those are secondary once the central theoretical result fails. I therefore recommend rejecting the paper: the headline theoretical contribution is not supported by a valid derivation, and the empirical results cannot rescue a guarantee that was never established.","tokens_in":21747,"tokens_out":5002,"duration_ms":51297,"concrete_test":"Recompute the example immediately after Theorem 4.5 with the paper's stated values (Q0 row r=(0.7,0.3), B row r=(1,−1), α=0.081, γ=0.9, T=65). Using the proof's Eq. (14), compute Q_T(r,r)=0.7+0.081·(1−0.9^66)/0.1≈1.51 and Q_T(r,s)=0.3−0.081·(1−0.9^66)/0.1≈−0.51; verify that neither |Q_T(r,r)−1|≤0.01 nor |Q_T(r,s)|≤0.01 holds and that the row is not a probability distribution. If instead Eq. (1) is taken literally as Q_T=Q0+αγ^T B, then Q_T(r,r)≈0.7000 and no absorption occurs. Either reading falsifies Theorem 4.5 as stated. A secondary check: the exact α required to make the row's limiting values (1,0) is 0.03, while Eq. (12) only forces α>0.08, demonstrating the missing equality constraint.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing concern is not primarily the empirical bridge to real LLMs but the central theorem itself: Theorem 4.5 does not prove what it claims even for the toy Markov chain. The paper's Eq. (1) defines Q_t = Q0 + αγ^t B, but the proof's Eq. (14) instead uses Q_T(r,r)=Q0(r,r)+αΣ_{i=0}^T γ^i B(r,r). If Eq. (1) is literal, Eq. (14) is simply wrong. If the intended training update is cumulative, then the bound in Eq. (16) still does not follow: the term Q0(r,r)+αB(r,r)Σγ^i−1 equals [Q0(r,r)+αB(r,r)/(1−γ)−1] − αB(r,r)γ^{T+1}/(1−γ), and the first bracket is not controlled by the theorem's assumptions. In the paper's own example, α=0.081 is chosen because it exceeds the lower bound 0.08, but the value that centers the row is α=0.03; hence the raw limit is Q_T(r,r)≈1.51 and Q_T(r,s)≈−0.51, violating both claimed δ-bounds and producing entries outside [0,1]. The proof in Appendix G confirms this: Step 2 requires the exact equality α=(1−Q0(r,r))(1−γ)/B(r,r), not the lower bound stated in Eq. (12). Remark 4.6 and Algorithm 1 acknowledge values outside [0,1] and clip them, but that normalization is not part of Theorem 4.5, and the theorem asserts that QT is a transition matrix satisfying the bounds. Thus the foundational guarantee supporting safety depth, cyclic augmentation, and ensemble results is not actually derived.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper models autoregressive LLMs as Markov chains and claims a first theoretical result on safety alignment depth: Theorem 4.5 states that if the transition matrix is updated by adding a discounted bias αγ^t B with B(r,r)>0 and B(r,s)<0, then after T > log(δ(1-γ))/log(γ)-1 steps every refusal state r is δ-absorbing. It extends this to cyclic-group data augmentation (Proposition 4.8) and to ensemble aggregation (Proposition 4.11), and reports toy Markov-chain experiments plus fine-tuning of Gemma, Phi-2, and Qwen with shallow, deep, and cyclic augmentation. The paper itself acknowledges in Remark 4.6, Appendix F, and Appendix J that the raw updates can leave [0,1], that a normalization is applied in the experiments, and that the empirical study is proof-of-concept.","tokens_in":22169,"tokens_out":6997,"duration_ms":66565,"significance":"If Theorem 4.5 were correct, the paper would give a clean, falsifiable training-time formula for safety depth and a novel depth-width tradeoff for ensembles, and the toy experiments would be a useful sanity check. The breadth of the claim, presented as the first theoretical result on this question, makes correctness load-bearing. Unfortunately the central theorem is invalid as stated: the proof uses a cumulative update that contradicts the displayed update rule, the stated lower bound on α does not imply the claimed δ-absorbing bounds, and the normalization added afterward is outside the theorem. The cyclic and ensemble results inherit the same defect because they rely on the same update dynamics. The conceptual framing of refusal states as absorbing, and the ensemble aggregation formulas themselves, have some merit, but they do not compensate for the missing central derivation.","major_comments":[{"comment":"The theorem's update rule is Qt = Q0 + αγ^t B, so QT(r,r) would be Q0(r,r) + αγ^T B(r,r); the proof instead writes QT(r,r) = Q0(r,r) + α Σ_{i=0}^T γ^i B(r,r). These are different processes. If Eq. (1) is intended literally, Eq. (14) is simply wrong; if the intended update is cumulative, that must be stated and the displayed Qt must be changed. As written, the proof does not prove the theorem.","section":"Section 4, Theorem 4.5, Eq. (1) vs. Eq. (14)"},{"comment":"Even under the cumulative interpretation, the claimed δ-absorbing bounds fail on the paper's own numbers. For Q0(r,r)=0.7, Q0(r,s)=0.3, B(r,r)=1, B(r,s)=-1, α=0.081, γ=0.9, T=65, the cumulative formulas yield Q_T(r,r)≈1.509 and Q_T(r,s)≈-0.509, so |Q_T(r,r)-1|≈0.509 and |Q_T(r,s)|≈0.509, far exceeding δ=0.01, and the row does not sum to 1. Appendix G, Step 2 shows that centering the row requires the exact equality α=(1-Q0(r,r))(1-γ)/B(r,r)=0.03, not the lower bound α>0.08 in Eq. (12). Therefore Eq. (12) is insufficient and the theorem's guarantee is not derived.","section":"Section 4, Theorem 4.5, worked example"},{"comment":"The paper responds to the out-of-range values by clipping and normalizing rows, and Remark 4.6 says this normalization is used in all numerical experiments. This step is not part of Theorem 4.5, and no proof is given that the clipped and normalized matrix satisfies the δ-absorbing bounds in Eq. (13), remains stochastic, or corresponds to a meaningful training process. Appendix F states only that the behavior is 'satisfied in practice.' Since the theorem's conclusion concerns QT itself, this post-hoc normalization cannot patch the proof.","section":"Section 4, Remark 4.6 and Appendix F, Algorithm 1"},{"comment":"The bridge from the toy Markov chain to LLM fine-tuning is assumed rather than established: Assumption 3.1 asserts that fine-tuning is equivalent to the iterative transition-matrix update, and Assumption 3.4 asserts that group-augmented fine-tuning is equivalent to conjugating the bias matrix B. No derivation, reference, or empirical check supports these equivalences, and Appendix J concedes that the analysis is behavior-level and proof-of-concept. Even if the toy theorem were correct, the LLM safety-depth conclusions would rest on unvalidated assumptions.","section":"Section 3, Assumptions 3.1 and 3.4"}],"minor_comments":[{"comment":"The phrase 'laerning rate' should be 'learning rate'.","section":"Section 4, Theorem 4.5 statement"},{"comment":"The sign conventions in lines 4-6 are inconsistent: eQT(i,s) is defined with a leading minus sign and Q+ is defined as -max{0,eQT(i,s)}, so the algorithm as printed does not compute the clipped raw update described in the surrounding text.","section":"Appendix F, Algorithm 1"},{"comment":"The proof headings refer to 'Theorem 4.8' and 'Theorem 4.10', but the statements being proved are Proposition 4.8 and Corollary 4.10.","section":"Appendix G, G.3 and G.4"},{"comment":"The example reports Q3(r,r)=1.0 and Q3(r,s)=0.0 exactly after three steps, but with α=0.081 and γ=0.9 the raw cumulative update gives Q3(r,r)≈0.819 and Q3(r,s)≈0.181; the displayed exact values can only come from the unstated normalization of Remark 4.6, which is not part of Proposition 4.8.","section":"Section 4, Example of Proposition 4.8"},{"comment":"The toy experiments set α=γ=1, but Theorem 4.5 requires γ∈(0,1) for the geometric-series argument and the logarithmic bound on T, so the toy experiment does not instantiate the theorem's assumptions.","section":"Section 5, Numerical Experiments"},{"comment":"The limitations section appropriately concedes that the paper treats the LLM at a behavior level and shows only proof-of-concept experiments; this is candid, but it also confirms that the LLM experiments do not close the gap created by Assumptions 3.1 and 3.4.","section":"Appendix J, Limitations"}],"recommendation":"reject","confidential_remarks":"The central theorem is not merely missing a step; the paper's own example and Appendix G show that the stated α condition is insufficient, and Remark 4.6 plus Algorithm 1 patch the result outside the theorem. The high-level idea of analyzing safety depth through absorbing states may be worth pursuing, but a correct version would need a genuinely stochastic update rule or a constrained-bias theorem, together with validation of Assumptions 3.1 and 3.4. I do not see a route to acceptance within the current manuscript, though I would be open to a resubmission with a corrected main theorem."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's idea is worth a look, but the central guarantee doesn't hold as stated, and the connection to real LLM fine-tuning is asserted, not derived. I agree with the reader's rejection.\n\nThe authors try to formalize \"safety depth\" by viewing an autoregressive LLM as a Markov chain and asking how many fine-tuning steps make a refusal state nearly absorbing. That framing is sensible, and the ambition to give a training-step bound is the kind of thing practitioners would value. The cyclic-group augmentation idea is also a nice twist, and the ensemble-depth/width tradeoff is a reasonable topic to explore. The authors do engage with the relevant literature (Zekri et al., Qi et al.), and they are honest in Appendix J that the work is behavior-level proof-of-concept.\n\nBut the mathematics falls apart at the main theorem. Equation (14) in the proof of Theorem 4.5 does not follow from the update rule in Eq. (1), and with the paper's own example numbers (Q0 row (0.7,0.3), α=0.081, γ=0.9, T=65) the raw update gives Q_T(r,r)≈1.51 and Q_T(r,s)≈−0.51, both outside [0,1]. That violates the claimed δ-absorbing bounds before any normalization. The authors try to patch this with clipping in Remark 4.6 and Algorithm 1, but that normalization is not part of the theorem, and the theorem asserts QT is a transition matrix satisfying the bounds. So Theorem 4.5 is invalid as a mathematical statement. Assumptions 3.1 and 3.4 are load-bearing and simply assert that fine-tuning and group-augmented fine-tuning follow the toy update; no derivation or independent validation is given. If those assumptions fail, the safety-depths conclusions do not transfer to LLMs. The toy experiment in Figure 3 uses α=γ=1, outside the theorem's 0<γ<1 hypothesis, so it doesn't validate the theorem either. Proposition 4.8's proof contains a false identity (ΣP^tBP^{-t}=mB). The ensemble proposition is essentially a union bound, Markov, and Hoeffding restatement, not a novel theoretical contribution. The LLM experiments measure safety scores, not any Markov-chain quantity, and the ensemble comparison picks Qwen 2.5 as the \"deep\" baseline because it performed best, which is a selection bias.\n\nIn short, the framing is suggestive but the core result is not sound. A serious editor should not spend referee time on the current version. If the authors correct Theorem 4.5 and provide some validation for Assumptions 3.1 and 3.4, the idea might become salvageable as a heuristic. I would not cite this in my own work yet, though it might make a useful reading-group example of how toy models can mislead.","headline":"Interesting framing undercut by an invalid central theorem and an unsupported bridge to real LLMs.","tokens_in":22780,"tokens_out":2737,"would_cite":false,"duration_ms":28375,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that repeated fine-tuning can make an LLM's refusal behavior permanent, with a closed-form step count for when that happens, and that wider ensembles can substitute for deeper alignment.","keywords":["safety alignment depth","Markov chain","refusal state","cyclic group augmentation","ensemble methods","LLM fine-tuning","safety guarantees"],"falsifier":"Estimate a real model's refusal-state row from token-generation statistics during the paper's fine-tuning recipe: if the escape probability $1 - Q_t(r,r)$ does not decay geometrically and drop below $\\delta$ at the predicted threshold $T > \\log(\\delta(1-\\gamma))/\\log(\\gamma) - 1$, or if the refusal row of the fitted empirical chain fails to satisfy $|Q_T(r,r)-1| \\leq \\delta$, then the toy bound is not what actual training does; the cheapest check is a replication on a small model such as Gemma 2B under the paper's published settings.","tokens_in":2039,"feed_emoji":"🛡️","tokens_out":5049,"duration_ms":140013,"temperature":0.7,"pith_summary":"The paper tries to answer a concrete engineering question: when safety alignment is inserted at a chosen output depth, how deep must that insertion be for the refusal to actually hold? The authors model an autoregressive LLM as a Markov chain and claim that fine-tuning with a small bias in favor of the refusal state drives that state to become effectively absorbing, meaning permanent refusal, after a number of steps given by a closed-form logarithm bound. They further claim that rotating the refusal phrase through training positions via a cyclic-group augmentation preserves the guarantee and can accelerate convergence, and that an ensemble of shallowly aligned models can jointly meet a safety threshold that no single shallow model meets alone. A sympathetic reader would care because these results convert a rule-of-thumb alignment heuristic into a computable design choice with stated step counts and per-model risk budgets.","feed_headline":"A closed-form bound sets when LLM refusals stick","feed_subtitle":"The bound says a refusal state becomes effectively absorbing after a computable number of fine-tuning steps.","key_machinery":"The load-bearing object is the discounted additive update $Q_t = Q_0 + \\alpha\\gamma^t B$, proposed in Assumption 3.1 as the model of fine-tuning: each step adds a bias matrix $B$, scaled by learning rate $\\alpha$ and geometrically discounted by $\\gamma^t$, where the refusal state $r$ receives positive self-bias $B(r,r)>0$ and negative escape-bias $B(r,s)<0$. The proofs hinge on the geometric-series identity $\\sum_{t=0}^T \\gamma^t = (1-\\gamma^{T+1})/(1-\\gamma)$, whose tail $\\gamma^{T+1}/(1-\\gamma) \\leq \\delta$ yields the explicit step threshold; the refusal state behaves as a near-absorbing state once the diagonal bias dominates its row. Assumptions 3.1 and 3.4, that fine-tuning equals this matrix update and that group-augmented fine-tuning equals conjugating $B$ by a permutation matrix, are the bridge from the toy chain to real LLM training, and the paper's own limitations section acknowledges that the analysis operates at a behavior level that may not cover every training dynamic.","core_discovery":"The central claim is Theorem 4.5 (the $\\delta$-absorbing theorem): in a Markov chain updated as $Q_t = Q_0 + \\alpha\\gamma^t B$, if the bias matrix satisfies $B(r,r)>0$ and $B(r,s)<0$ at a refusal state $r$, then for any tolerance $\\delta$ the state $r$ becomes $\\delta$-absorbing once the training steps exceed $T > \\log(\\delta(1-\\gamma))/\\log(\\gamma) - 1$ and the learning rate exceeds $\\max_{r,s}|Q_0(r,s)|(1-\\gamma)/\\min(B(r,r), -B(r,s))$; at that point $|Q_T(r,r)-1| \\leq \\delta$ and $|Q_T(r,s)| \\leq \\delta$ for every non-refusal state $s$. Corollary 4.7 turns this into a per-state threshold for the largest safety depth $r^*$ that becomes absorbing given a chosen pair $(\\alpha, T)$. Proposition 4.8 shows the guarantee survives time-varying bias of the form $B(t) = P^t B P^{-t}$ coming from a cyclic permutation, with the step bound governed by the permutation order $\\mathrm{ord}(P)$. Proposition 4.11 distributes an overall safety level $\\varepsilon$ across an ensemble of width $W$: per-model risk must be at most $\\varepsilon/W$ under union aggregation, $\\varepsilon\\tau$ under averaging with threshold $\\tau$, or $\\tfrac12 - \\sqrt{\\ln(1/\\varepsilon)/(2W)}$ under majority voting, so broadening the ensemble relaxes each model's training requirement.","pith_inferences":["A natural testable extension the paper does not run: use the closed-form step bound to predict when a real model's refusal survives adversarial jailbreak prompts, since the Markov abstraction drops attention heads and positional structure that may control where refusal generalizes.","The ensemble results imply a resource trade-off the paper leaves unpriced: union aggregation requires $W$ models trained to $\\varepsilon/W$, roughly $W$ times the compute of one deep model, so majority voting is the only strategy that clearly makes widening cheaper than deepening; an explicit cost comparison would settle which strategy practitioners should prefer.","Because Proposition 4.8 only needs the bias sequence to be bounded and periodic under permutation conjugation, the guarantee should carry over to any finite group of dataset permutations, not just the cyclic rotations the paper instantiates."],"forward_implications":["Safety depth becomes a computed quantity rather than a tuning guess: given the discount $\\gamma$, tolerance $\\delta$, initial matrix $Q_0$, bias $B$, and learning rate $\\alpha$, the required step count follows from a closed-form log bound, and the deepest refusal state that is guaranteed $\\delta$-absorbing is $r^* = \\max\\{r \\in R \\mid \\alpha > \\alpha_r,\\ T > T_r\\}$.","Cyclic-group data augmentation, which rotates the refusal phrase through training positions, preserves the $\\delta$-absorbing guarantee under a periodically varying bias and can converge in fewer steps, with the bound capped by the permutation order $\\mathrm{ord}(P)$.","Ensemble width trades against alignment depth: $W$ models each trained to per-model risk $\\varepsilon/W$ (union), $\\varepsilon\\tau$ (average), or $\\tfrac12 - \\sqrt{\\ln(1/\\varepsilon)/(2W)}$ (majority voting) collectively meet the $\\varepsilon$-safety threshold, reducing the training burden on any single model.","Experiments on Gemma 2B, Phi-2 2B, and Qwen 2.5 1.5B report cyclic augmentation raising mean safety scores, with Gemma rising from 0.42 (shallow) and 0.46 (deep) to 0.61 (cyclic), and ensembles of three shallow models scoring higher and less variably than a single deep model."],"supporting_citations":[{"why":"Supplies the equivalence between autoregressive LLMs and Markov chains on which the whole safety-depth analysis is built.","marker":"Zekri et al., 2024"},{"why":"Introduces shallow versus deep safety alignment and the depth-insertion data augmentation that this paper formalizes and extends with cyclic groups.","marker":"Qi et al., 2024"},{"why":"Documents that fine-tuning aligned models compromises safety, the motivating failure this paper addresses.","marker":"Qi et al., 2023b"},{"why":"Provides the MaliciousInstruct training set used in the open-source LLM experiments.","marker":"Huang et al., 2024"},{"why":"Supplies the union bound, Hoeffding, and Chernoff inequalities behind the ensemble safety bounds in Proposition 4.11.","marker":"Boucheron et al., 2003"},{"why":"The neural tangent kernel lens that the paper explicitly contrasts with its Markov-chain analysis in the discussion.","marker":"Jacot et al., 2018"}],"fun_headline_variants":["Markov chain theorem sets LLM refusal depth","New proof bounds when LLM refusals become absorbing","Alignment depth vs ensemble width: theory resolves tradeoff","First theory pinpoints safe alignment depth in LLMs"],"cache_read_input_tokens":24576,"weakest_assumption_plain":"The load-bearing premise is that fine-tuning an LLM really is the simple discounted additive update $Q_t = Q_0 + \\alpha\\gamma^t B$, and that rotating the training data is equivalent to conjugating that bias matrix; the paper asserts these identifications without deriving them from gradient dynamics, and it states in its own limitations section that it treats LLMs at a behavior level that may not cover every case.","fun_headline_variants_meta":{"raw":{"variants":["Markov chain theorem sets LLM refusal depth","New proof bounds when LLM refusals become absorbing","Alignment depth vs ensemble width: theory resolves tradeoff","First theory pinpoints safe alignment depth in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1625,"prompt_tokens":1050,"completion_tokens":575,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":666,"tokens_out":575,"duration_ms":6662,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T18:09:22.264658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate a real model's refusal-state row from token-generation statistics during the paper's fine-tuning recipe: if the escape probability $1 - Q_t(r,r)$ does not decay geometrically and drop below $\\delta$ at the predicted threshold $T > \\log(\\delta(1-\\gamma))/\\log(\\gamma) - 1$, or if the refusal row of the fitted empirical chain fails to satisfy $|Q_T(r,r)-1| \\leq \\delta$, then the toy bound is not what actual training does; the cheapest check is a replication on a small model such as Gemma 2B under the paper's published settings.","supporting_citations":[{"cited_title":"Catastrophic jailbreak of open-source llms via exploiting generation","cited_arxiv_id":null,"evidence_quote":"Provides the MaliciousInstruct training set used in the open-source LLM experiments."},{"cited_title":"Concentration inequalities","cited_arxiv_id":null,"evidence_quote":"Supplies the union bound, Hoeffding, and Chernoff inequalities behind the ensemble safety bounds in Proposition 4.11."}],"review_version":1}