{"id":"d21aac99-a062-41e4-8750-b9c3bd865836","arxiv_id":"2610.19072","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HERD uses a copy-weighted delta-closure on a typed lineage graph to bound cumulative residual in federated unlearning under mutual distillation, achieving 18-52% less residual at 6-12% of baseline cost.","lead":"Federated unlearning under mutual distillation suffers from 'resurrection' where forgotten data re-enters the teacher via student uploads. HERD selects a minimal set of holders to correct by tracking how many times each residual will be copied through the typed knowledge lineage graph.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Assumption 1 (independent W[t]) is load-bearing: Theorem 1’s Neumann series argument fails under correlated matrices, leaving Eq. (8) unvalidated even when ρ(¯W)<1.","rationale":"The reader’s weakest_assumption correctly identifies Assumption 1 as the most fragile point. The stress-test confirms that this is the single most load-bearing concern: without independence, the copy weights derived from the Neumann series are not guaranteed to bound the expected cumulative residual. However, the paper is transparent about this assumption, states it clearly, and builds the theory on it. The method is novel, the experimental results on TOFU show practical improvement, and the authors acknowledge that the inheritance matrix is not fixed. Therefore, the concern does not warrant changing the ACCEPT verdict; rather it motivates a concrete sensitivity check that a follow-up could address. The verdict stays unchanged because the paper delivers its claimed contributions within the stated assumptions.","tokens_in":16177,"tokens_out":3301,"duration_ms":37683,"concrete_test":"Simulate N=50 clients over 20 rounds with mutual distillation. Let each client’s link state follow an AR(1) process on the logit scale (autocorrelation ρ=0.3). Let the fraction of distillation prompts that touch u follow a two-state Markov chain with transition probability 0.8 per round (so excitation correlates over time). Fit ¯W from the marginal means, compute φ and select K via HERD. For 100 random seeds, record the actual cumulative residual after correction and compare its mean and 95th percentile to φ^⊤ r+. If the mean exceeds φ^⊤ r+ by more than 10%, or if the 95th percentile exceeds 1.5×φ^⊤ r+, then the independence assumption is materially consequential for the method’s guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The entire theoretical guarantee that partial correction suffices when ρ(¯W)<1 rests on Assumption 1: the inheritance matrices W[t] are independent across rounds and independent of r[t]. Theorem 1 then uses the Neumann series of ¯W to equate the expected cumulative residual to φ^⊤ r+. If W[t] are not independent, the expectation of the infinite product Π_{s≥t_u} W[s] does not factor into powers of ¯W, and Eq. (8) is no longer guaranteed to bound even the expected cumulative residual. The paper acknowledges that the inheritance matrix is not fixed (sentence [b76]) but does not weaken the independence condition. Real federated systems exhibit correlated client participation (e.g., bursty link outages) and correlated distillation prompts, making the independence assumption fragile. A practical failure would be that selected δ-closure leaves more residual than predicted, or that the cumulative residual exceeds δ even when ρ(¯W)<1. This is a genuine soft spot because the method’s correctness depends on it, and the paper offers no robustness analysis or concentration bound.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes HERD, a method for federated unlearning under mutual distillation where a removal request on the teacher alone can be undone by student uploads. HERD records a typed knowledge lineage graph, computes copy weights (expected total exposure from a residual) via a Neumann series of the expected inheritance matrix, and selects the cheapest set of holders to correct so that the cumulative residual after the request stays below a threshold δ. The theoretical analysis (Theorem 1, Propositions 1–3) gives conditions under which partial correction suffices (spectral radius of the expected inheritance matrix below one) and provides a rejoin rule for offline clients. Experiments on the TOFU dataset with simulated and hardware testbeds show that HERD leaves 18–52% less cumulative residual than the strongest baseline at 6–12% of its correction cost.","tokens_in":16497,"tokens_out":2240,"duration_ms":24561,"significance":"If the results hold, HERD addresses a genuine gap in federated unlearning: the resurrection of removed knowledge through mutual distillation loops that cannot be solved by correcting only the teacher or all descendants. The idea of using copy weights derived from the server's own records, without sending the removed data to clients, is novel and practical. The theoretical characterization of when partial correction suffices (spectral radius condition) and the proof that averaging alone never reaches a spectral radius of one are clean insights.","major_comments":[{"comment":"Proposition 1 states that the guard brings the spectral radius below one when traits spread weakly within a family, as measured by ν. Yet the paper never reports the fitted values of κ_x, κ_s, and ν, nor the resulting spectral radius ρ(ar{W}) before and after the guard, in the TOFU experiments. Without this information, it is unclear whether the condition ρ(ar{W})<1 actually holds in the evaluated settings. If it does not, then the theoretical guarantee for partial correction (Theorem 1) would not apply, and the empirical success would lack a theoretical basis. Please report these values and discuss whether the guard actually achieves ρ(ar{W})<1 for the TOFU federation.","section":"Sec. IV, Assumption 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The paper uses 'copy number' to refer to the spectral radius ρ(ar{W}) and 'copy weight' for φ_j. While the distinction is clear in context, a definitional sentence at the end of Section III-B would help avoid confusion.","section":"General","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The caption lists 'score' values that are not explicitly defined in the figure or caption. Please add a brief explanation of what the numbers represent (e.g., copy weight or hop weight) and how they are computed for this illustration.","section":"Fig. 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The testbed results use only two clients (one same-family, one cross-family), while the simulation uses 50 clients. The paper should note that the hardware numbers are illustrative and may not reflect the scaling behavior shown in Fig. 7a.","section":"Table I","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The simulation sets δ as a percentage of uncorrected exposure (Exp). The paper should clarify the relationship between the budget δ in Problem (P) and the exposure reported in Table I (i.e., the absolute residual bound vs. the percentage metric).","section":"Sec. V-A","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Shannon capacity formula in Eq. (1) assumes no interference; this is a standard simplification but should be stated as an assumption.","section":"Eq. (1)","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Thanks for sending this. The paper addresses an important gap: federated unlearning when the server and devices train by mutual distillation. Existing methods either ignore the return path from students back to the teacher (resurrection) or correct all descendants at high cost. HERD formalizes the problem as selecting a minimal-cost set of holders whose correction bounds the cumulative residual, using a typed lineage graph with copy weights that capture whether averaging or distillation spreads the residual.\n\nThe core insight—that averaging alone never pushes the spectral radius of the expected inheritance matrix past one, while distillation can—is sound and practically useful. The experiments on TOFU are convincing: HERD leaves 18–52% less cumulative residual than UNDO at 6–12% of its correction cost, across client, class, and sample unlearning scenarios. The theoretical statements (Theorem 1, Propositions 1–3) hold under the stated assumptions.\n\nNow the soft spot. The stress-test concern is real: Assumption 1 (independence of W[t] across rounds) is load-bearing for Theorem 1. The Neumann series argument that E[Σ r[t]] = φᵀ r₊ depends on factoring powers of the mean matrix, which fails when W[t] are correlated. The authors acknowledge that the inheritance matrix is not fixed (sentence [b76]) but do not relax independence. In practice, client participation, link quality, and distillation prompts are correlated over time, so the guarantee might degrade. The paper offers no concentration bound or sensitivity analysis. This is a genuine limitation, not a minor one, because the method's correctness depends on ρ(W̅) < 1 being the right threshold under correlation.\n\nMinor concern: the fitted parameters κₓ, κₛ, ν via canaries introduce calibration dependence, but the authors are upfront about it and it's a practical necessity.\n\nOverall, the paper makes a solid contribution to a problem that matters. The fragility of Assumption 1 should be addressed in revision—either by proving a bound under mixing or by providing empirical robustness checks. I would accept a peer-review invitation and send it to referees, with the independence assumption as the primary discussion point. The work is worth engaging.","headline":"HERD solves a real problem (resurrection under mutual distillation) with a clean formulation and order-of-magnitude cost savings, but the load-bearing independence assumption (Assumption 1) is fragile and the paper offers no robustness analysis.","tokens_in":17027,"tokens_out":1082,"duration_ms":12966},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"HERD selects the cheapest set of holders to correct by bounding the cumulative residual through copy weights, proving that partial correction works only when the spectral radius of the expected inheritance matrix is below one.","keywords":["Federated unlearning","Knowledge distillation","Large language models","Edge intelligence","Knowledge lineage","Copy number","Resurrection","Mutual distillation"],"falsifier":"Construct a federated mutual-distillation experiment with a copy number ρ(W̄) just below 1 (e.g., 0.98) but where the inheritance matrices are strongly correlated across rounds (e.g., the same set of clients is offline for several consecutive rounds). If the cumulative residual after correction diverges or fails to decay as predicted by the Neumann series, the independence assumption is violated and the theoretical guarantee no longer holds.","tokens_in":17627,"feed_emoji":"🧹","tokens_out":2773,"duration_ms":31177,"temperature":0.7,"pith_summary":"Under mutual distillation in federated learning, deleting data from the teacher model is not enough because the knowledge has been copied into student models and will be taught back. The paper proposes HERD, which records a typed knowledge lineage graph from transfers the server already performs and computes a 'copy weight' for each holder—the total future exposure that one unit of residual creates. HERD then selects the cheapest set of holders whose correction keeps the cumulative residual below a threshold. The central theoretical result is that partial correction suffices only when the copy number (spectral radius of the expected inheritance matrix) is below one, and that averaging alone never reaches one. Experiments on TOFU show HERD leaves 18% to 52% less cumulative residual than the strongest baseline at 6% to 12% of its correction cost.","feed_headline":"Copy number below 1 enables partial erasure in federated learning","feed_subtitle":"HERD selects the cheapest holders to correct, cutting residual by up to 52% at 12% cost via a lineage graph.","key_machinery":"The copy number is the spectral radius ρ(W̄) of the expected inheritance matrix W̄, where each entry w̄ᵢⱼ is the expected fraction of holder j's residual that holder i inherits in one round. The copy weight vector φ = (I − W̄ᵀ)⁻¹ 1 gives the total exposure created by one unit of residual at each holder. The typed lineage graph records edges weighted by transfer type: averaging rows sum to one (spreads residual), distillation rows sum to more than one (copies residual). The selection problem is a covering knapsack: minimize correction cost subject to Σⱼ φⱼ rⱼ[tᵤ] ≤ δ for the uncorrected holders.","core_discovery":"The paper identifies and formalizes the 'resurrection' problem in federated unlearning under mutual distillation: after correcting the teacher, student uploads still carry the removed data's effect and teach it back. To solve this, HERD records every transfer's type (averaging or distillation) into a lineage graph without extra communication, then computes copy weights that account for the entire forward pathway including return edges. The core condition is that partial correction is viable only when the copy number ρ(W̄) < 1; if ρ(W̄) ≥ 1, any nonzero residual leads to unbounded cumulative exposure. The paper further proves that averaging alone never reaches ρ = 1, so distillation is the sole cause of resurrection.","pith_inferences":["The paper's independence assumption on inheritance matrices (Assumption 1) may be violated in real systems where client participation and link quality correlate over time; a natural extension would be to derive bounds under Markovian or adversarial perturbations.","Because the copy weight φ = (I − W̄ᵀ)⁻¹ 1 depends on the entire graph, a single student with high out-degree (e.g., a strong same-family absorber) can force the closure to include many holders; scaling analysis for large networks would be valuable.","The technique of using canaries to measure absorption rates κₓ, κₛ, and ν is clever but assumes canaries are representative of u; a robustness check against canary mismatch would strengthen the practical deployment.","The method's applicability extends beyond LLMs to any federated system with heterogeneous models exchanging soft targets, such as vision transformers or recommendation models, as long as the transfers pass through a central server."],"forward_implications":["Federated unlearning systems that use mutual distillation must account for the full distillation loop, not just the teacher model, otherwise the removed data resurfaces within a few rounds.","The copy number condition ρ < 1 provides a simple necessary and sufficient test for whether partial correction is feasible; systems can monitor this number online to decide their unlearning strategy.","Averaging alone cannot cause resurrection because every row of an averaging matrix sums to one, keeping the spectral radius below one after scaling by the retention factor λⱼ < 1.","The transfer guard (dropping outputs that carry u) can reduce ρ below one even when distillation edges would otherwise push it above one, but the floor ν of same-family distillation may still allow weak copying.","The rejoin rule for offline holders uses the round of their last transfer to estimate residual, and corrects them before their first upload if their copy weight exceeds the remaining budget, keeping the closure valid."],"supporting_citations":[{"why":"SimNPO: baseline that corrects only the teacher, representing the naive approach that ignores resurrection.","marker":"[27]"},{"why":"FIUn: baseline that corrects every descendant of u in an unweighted lineage graph, providing a comparison for lineage-based approaches.","marker":"[30]"},{"why":"DU: baseline that sends a Newton-style correction to the same holders, used as a stronger baseline.","marker":"[22]"},{"why":"UNDO: baseline that noises descendants and distills them again from the corrected teacher, shown to be the strongest prior work.","marker":"[32]"},{"why":"PURGE: baseline that shards data before training to map teacher shards to student shards, representing a sharding strategy.","marker":"[31]"},{"why":"Cloud et al. (2026) provides the evidence for the floor ν in same-family distillation, which HERD uses to upper-bound post-guard copy number.","marker":"[33]"}],"fun_headline_variants":["Copy number below 1 makes partial unlearning possible","HERD cuts unlearning residual by 52% at 12% cost","Distillation causes unlearning resurrection; lineage graph fixes it","Unlearning resurrection solved: copy number must stay below 1","Federated unlearning: averaging safe, distillation resurrects data"],"cache_read_input_tokens":0,"weakest_assumption_plain":"Assumption 1 requires that the inheritance matrices W[t] are independent across rounds and independent of r[t], and that their mean W̄ is nonnegative and irreducible; in real federated systems where client participation, link quality, and distillation prompts correlate over time, this independence may fail and irreducibility may hold only in expectation over long periods.","fun_headline_variants_meta":{"raw":{"variants":["Copy number below 1 makes partial unlearning possible","HERD cuts unlearning residual by 52% at 12% cost","Distillation causes unlearning resurrection; lineage graph fixes it","Unlearning resurrection solved: copy number must stay below 1","Federated unlearning: averaging safe, distillation resurrects data"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":9.3e-05,"raw_usage":{"cost":9.336824e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":9.336824e-05,"upstream_inference_prompt_cost":3.49076e-05,"upstream_inference_completions_cost":5.846064e-05},"total_tokens":1672,"prompt_tokens":910,"completion_tokens":762,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":674}},"tokens_in":910,"tokens_out":762,"cost_source":"openrouter_usage","duration_ms":9490,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":674,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-10-15T16:56:46.619504+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Construct a federated mutual-distillation experiment with a copy number ρ(W̄) just below 1 (e.g., 0.98) but where the inheritance matrices are strongly correlated across rounds (e.g., the same set of clients is offline for several consecutive rounds). If the cumulative residual after correction diverges or fails to decay as predicted by the Neumann series, the independence assumption is violated and the theoretical guarantee no longer holds.","supporting_citations":[{"cited_title":"Sim- plicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning,","cited_arxiv_id":null,"evidence_quote":"SimNPO: baseline that corrects only the teacher, representing the naive approach that ignores resurrection."},{"cited_title":"Parallel Unlearn- ing in Inherited Model Networks,","cited_arxiv_id":null,"evidence_quote":"FIUn: baseline that corrects every descendant of u in an unweighted lineage graph, providing a comparison for lineage-based approaches."},{"cited_title":"Certified Unlearning in Decentralized Federated Learning,","cited_arxiv_id":null,"evidence_quote":"DU: baseline that sends a Newton-style correction to the same holders, used as a stronger baseline."},{"cited_title":"Distillation Robustifies Unlearning,","cited_arxiv_id":null,"evidence_quote":"UNDO: baseline that noises descendants and distills them again from the corrected teacher, shown to be the strongest prior work."},{"cited_title":"Efficient Verified Unlearning For Distillation,","cited_arxiv_id":null,"evidence_quote":"PURGE: baseline that shards data before training to map teacher shards to student shards, representing a sharding strategy."},{"cited_title":"Language models transmit behavioural traits through hidden signals in data,","cited_arxiv_id":null,"evidence_quote":"Cloud et al. (2026) provides the evidence for the floor ν in same-family distillation, which HERD uses to upper-bound post-guard copy number."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}