{"id":"4f4b3060-59b2-4076-b620-235f5ae125cd","arxiv_id":"2607.08337","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Cross-attention maps serve as a tractable surrogate for manifold proximity, enabling automatic synthesis of anchors that suppress normal-space drift in diffusion unlearning.","lead":"AutoAnchor automatically builds manifold-proximal anchors for diffusion unlearning by optimizing a cross-attention consistency loss, replacing manual or empty-prompt directions. This yields more stable concept erasure and better utility preservation on text-to-image models, and can plug into existing unlearning algorithms.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Assumption 5 is the sole bridge from attention loss to ζ-proximity, yet the paper never measures whether attention-map distance actually bounds normal-space score residual.","rationale":"The Reader correctly isolates Assumption 5 as the single load-bearing bridge. Theorems 1–5 are standard high-dimensional concentration arguments once the proximal-anchor definition is granted; the empirical tables show consistent gains when AutoAnchor is plugged into seven baselines. The only place the argument can still break is the unmeasured isomorphism that converts attention consistency into normal-drift control. Because that link is neither proved nor empirically calibrated, the geometric explanation for the gains remains conditional, matching the Reader’s verdict. No stronger internal inconsistency appears; the concern is precisely the one already flagged.","tokens_in":33769,"tokens_out":584,"duration_ms":6971,"concrete_test":"On a frozen SD-v1.4 U-Net, sample 500 (z_t,t) pairs under a target concept (e.g., Mickey Mouse). For each of 50 random anchors spanning a range of LCA values, compute both LCA and a Monte-Carlo estimate of the normal residual (project the score difference onto the estimated normal space via local PCA of nearby latents, or measure Euclidean distance of reverse trajectories from a high-quality manifold proxy). Report the Pearson correlation and the empirical Lipschitz ratio. If correlation <0.4 or the ratio is unbounded, Assumption 5 fails and the ζ-proximity claim is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that AutoAnchor produces ζ-proximal anchors (and thereby suppresses normal drift per Theorems 2/6) rests entirely on Assumption 5 (Appendix A.7): ||Π_N(ε_\theta*(z_t,e_u)-ε_\theta*(z_t,e_a))|| ≤ L_CA ∑_l ||A_l(u)-A_l(a)||_F + \nu. This is the only link that lets the tractable LCA objective control the geometric quantity ζ required by Definition 1 and Theorem 6. The paper supplies neither an empirical estimate of L_CA/\nu nor a correlation between LCA and any proxy for normal residual (e.g., off-manifold distance of reverse trajectories, or the fraction ||Δ_N||^{2}/||Δ||^{2}). Without that measurement, the theoretical guarantee that the optimized anchor is manifold-proximal remains an untested architectural hypothesis; the observed CLIP gains could arise from better semantic alternatives alone rather than from the claimed geometric mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that diffusion unlearning is unstable because anchor-free updates and manually chosen semantic anchors induce large normal-space drift off the data manifold. Under high-dimensional concentration assumptions it proves that such updates concentrate almost all energy in the normal space (Theorems 1, 3), while a ζ-proximal anchor suppresses that drift (Theorems 2, 4–5). To construct such anchors it proposes AutoAnchor: Stage I automatically generates and filters LLM candidates; Stage II optimizes a cross-attention consistency loss LCA as a tractable surrogate for manifold proximity (Assumption 5 / Theorem 6). Empirically, AutoAnchor improves concept removal and non-target utility on three primary and four additional targets, and when plugged into seven prior methods yields average gains of roughly 6% on CLIP(U) and CLIP(R).","tokens_in":34089,"tokens_out":1230,"duration_ms":19048,"significance":"If the geometric account holds, the work supplies a unified explanation for several known failure modes of diffusion unlearning (catastrophic forgetting, fragile erasure, biased redirection) and a lightweight, automatic preprocessing step that improves both anchor-free and anchor-based baselines without redesigning their losses. The plug-and-play empirical results across seven methods and multiple concept categories are a concrete contribution. The formal random-walk stability comparison and the explicit ζ-proximal definition are useful organizing tools even if the attention–geometry bridge remains partly conjectural. The contribution is therefore of clear interest to the machine-unlearning and generative-modeling communities, provided the load-bearing surrogate claim is better substantiated.","major_comments":[{"comment":"Appendix A.7, Assumption 5 and Theorem 6: the claim that optimizing LCA yields a ζ-proximal anchor rests entirely on the untested Attention–Geometry Isomorphism (normal residual Lipschitz in Frobenius attention distance plus tolerance ν). The manuscript never reports an estimate of L_CA or ν, nor any correlation between LCA and a geometric proxy for normal drift (e.g., ||Δ_N||²/||Δ||², off-manifold distance of reverse trajectories, or score residual projected onto an estimated normal space). Without that measurement, Theorems 2 and 6 do not yet certify that AutoAnchor anchors are manifold-proximal; the observed CLIP/FID gains could arise from better semantic alternatives alone. A short empirical check of this bridge is load-bearing for the central theoretical claim.","section":null},{"comment":"§4–§5 and Definition 1: the paper never measures the normal-energy fraction (or any related geometric quantity) for AutoAnchor versus empty-prompt or manual-anchor baselines. Table 2 shows that Stage-I semantic aggregation alone can degrade CLIP(U) while Stage-II LCA optimization recovers it, which is consistent with the story but does not directly test the normal-drift mechanism that Theorems 1–5 are written to explain. Reporting even a coarse proxy of ||Π_N(Δ)|| before and after optimization would substantially strengthen the link between theory and method.","section":null},{"comment":"§5.2 and Table 1: when AutoAnchor is integrated into originally anchor-free methods (ESD, ResAlign, CCRT, tuneSD-AF), the paper replaces the unconditional term by the AutoAnchor embedding. The main text does not clearly state whether this changes the official loss formulation of those methods or only the conditioning embedding, nor whether hyper-parameters were re-tuned. Because the average “+6.30% / +6.65%” claim is a primary empirical selling point, the precise adaptation protocol should be specified so that the gains can be attributed to the anchor rather than to incidental loss changes.","section":null}],"minor_comments":[{"comment":"Figure 1 caption and body: the geodesic / Euclidean distinction is helpful, but the figure itself is dense; a short legend defining the three colored arrows would improve readability.","section":null},{"comment":"Eq. (1) and surrounding text: Δ_AF and Δ_AB are written as noise-prediction differences; later the discrete update multiplies by a learning rate η (Theorems 3–4). Stating the discrete parameter-update step explicitly would avoid ambiguity.","section":null},{"comment":"Table 1: Δ values ≤2% are grayed as “below error precision,” yet the abstract still quotes average gains that include those entries. Clarifying how averages are computed (with or without the gray cells) would help.","section":null},{"comment":"§5.1: the clustering function f_cluster and the automatic choice of M via silhouette score are only fully described in Appendix B.6; a one-sentence pointer in the main text would help reproducibility.","section":null},{"comment":"Appendix B.3–B.4: robustness to LLM choice and K is useful; stating the default (K, N, λ) used in the main tables in a single place would make the experimental protocol easier to follow.","section":null},{"comment":"Typos / polish: “unrobust” appears repeatedly (prefer “non-robust”); “Weboldthe” in Table 1 caption; occasional missing spaces around citations.","section":null}],"recommendation":"major_revision","confidential_remarks":"The empirical plug-in results look solid and are likely the part most useful to practitioners. The theory is carefully written once the assumptions are granted, but the untested Assumption 5 is the single point that keeps the geometric narrative from being fully closed. I would not reject on that basis; a major-revision request focused on measuring the attention–normal residual link (or on clearly demoting the geometric claim to a motivating hypothesis) should be sufficient. Scope is appropriate for a solid ML venue."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful core is the normal-space analysis. Under the manifold hypothesis they show that empty-prompt or biased-anchor updates put almost all energy into the normal directions (Theorems 1–3), while a ζ-proximal anchor keeps the update mostly tangential (Theorems 2, 4–5). That diagnosis is cleaner than the usual “catastrophic forgetting / utility drop” laundry list and explains why both anchor-free and manual-anchor methods keep failing in the same ways.\n\nWhat they actually ship is AutoAnchor: LLM candidates → cluster filter → average embedding → optimize a cross-attention consistency loss inside a semantic shell. The loss is a sensible architectural surrogate (attention maps control layout), the two-stage pipeline is automatic, and the empirical transfer is real: plugging the resulting anchor into ESD, ConAbl, FMN, CCRT, etc. improves CLIP(U) by roughly 3–6 % and CLIP(R)/FID by smaller but consistent margins across Mickey Mouse, Van Gogh, nudity, Porsche, Picasso and a few others. Ablations show that the semantic-only stage is not enough; the attention optimization is doing work. Overhead is modest (4–8 %).\n\nThe soft spot is exactly the one the stress-test flags. Assumption 5 (attention-map Frobenius distance Lipschitz-controls the normal residual of the score) is the only bridge from LCA to the geometric ζ that Theorems 2 and 6 care about. They never measure L_CA, ν, or any correlation between LCA and an off-manifold proxy. So the claim that the optimized anchors are manifold-proximal remains an architectural hypothesis; the CLIP gains could partly come from simply better semantic alternatives. That does not kill the paper—the theorems are still correctly derived from the stated assumptions, the method is still useful—but it means the geometric story is not yet closed.\n\nCode is not released, absolute erasure is still incomplete, and a handful of free parameters (K, radii, λ) exist, though they look reasonably robust. Citation pattern is fair; they engage the right prior work.\n\nThis is for people who already care about concept erasure in diffusion models. It deserves a serious referee. I would bring it to reading group and would cite the normal-drift framing and the transfer numbers if I were working in the area.","headline":"Clean geometric diagnosis of why diffusion unlearning drifts off-manifold, plus a practical automatic anchor that improves seven baselines; the attention-to-geometry bridge is assumed rather than measured.","tokens_in":34649,"tokens_out":577,"would_cite":true,"duration_ms":8644,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Stable diffusion unlearning needs a manifold-proximal anchor; cross-attention maps can synthesize one automatically.","keywords":["diffusion unlearning","manifold hypothesis","cross-attention","concept erasure","Stable Diffusion","anchor-based unlearning","normal-space drift"],"falsifier":"Construct a pair of embeddings whose cross-attention maps are nearly identical yet whose score residual still has a large normal component on the true data manifold; if such pairs routinely appear, the attention-geometry bridge fails and AutoAnchor’s stability claims collapse.","tokens_in":34670,"feed_emoji":"🎯","tokens_out":790,"duration_ms":8523,"temperature":0.7,"pith_summary":"Text-to-image diffusion models can be made to forget a concept, but the usual ways of choosing the update direction push the model off the low-dimensional manifold of valid images. The paper proves that any update lacking a nearby anchor on that manifold leaks almost all of its energy into the normal space, which produces fragile erasure and collateral damage to unrelated concepts. AutoAnchor builds the missing anchor in two automatic stages: it first gathers and filters candidate concepts, then optimizes a cross-attention consistency loss that acts as a cheap surrogate for manifold proximity. The resulting anchors give stronger target removal and better non-target utility, and they can be dropped into existing unlearning algorithms to improve them as well.","feed_headline":"Cross-attention builds the missing anchor for stable unlearning","feed_subtitle":"Manifold-proximal anchors cut target CLIP scores by up to 31% and lift utility on non-target concepts","key_machinery":"Cross-attention consistency loss: the expected Frobenius distance between the frozen U-Net’s cross-attention maps under the target embedding and under the candidate anchor embedding; minimizing it (inside a spherical semantic shell) is proved to bound the geometric normal-drift quantity ζ that Theorems 2 and 6 require for stability.","core_discovery":"Under the manifold hypothesis, unlearning updates without a manifold-proximal anchor concentrate almost all of their energy in the normal space and therefore leave the data manifold; a carefully constructed proximal anchor keeps the update nearly tangential, and a cross-attention consistency loss is a tractable surrogate that produces such anchors automatically.","pith_inferences":["If attention maps are only a loose proxy, stronger geometric surrogates (local PCA of the score, estimated reach) may still be needed for high-curvature concepts.","The same normal-drift analysis likely applies to unlearning in other score-based or flow-based generative models that live on low-dimensional manifolds.","Adversarial re-learning attacks that succeed after AutoAnchor would indicate residual normal components that the current loss did not fully suppress."],"forward_implications":["Any diffusion unlearning method whose update direction is defined by an empty or purely semantic prompt will systematically leak energy off-manifold.","Existing anchor-based algorithms can be upgraded simply by replacing their hand-chosen anchors with AutoAnchor embeddings.","Concept removal and utility preservation need not trade off once the update is forced to stay nearly tangential.","The same attention-based surrogate can be reused for other latent-space editing tasks that require manifold-proximal directions."],"fun_headline_variants":["Cross-attention surrogate auto-builds manifold anchors for stable unlearning","Manifold-proximal anchors via attention keep unlearning updates tangential","AutoAnchor uses cross-attention consistency to prevent normal-space drift","Cross-attention as manifold surrogate stabilizes diffusion concept unlearning","Proximal anchors from attention consistency cut target concepts without bias"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The paper assumes that how far an update pushes a latent off the image manifold is controlled by how differently the target and the anchor light up the model’s cross-attention maps; if that link is weak, the surrogate no longer guarantees a stable anchor.","fun_headline_variants_meta":{"raw":{"variants":["Cross-attention surrogate auto-builds manifold anchors for stable unlearning","Manifold-proximal anchors via attention keep unlearning updates tangential","AutoAnchor uses cross-attention consistency to prevent normal-space drift","Cross-attention as manifold surrogate stabilizes diffusion concept unlearning","Proximal anchors from attention consistency cut target concepts without bias"]},"model":"grok-4.5","effort":"low","cost_usd":0.007426,"raw_usage":{"total_tokens":1845,"prompt_tokens":815,"num_sources_used":0,"completion_tokens":90,"cost_in_usd_ticks":74260000,"prompt_tokens_details":{"text_tokens":815,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":940,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":815,"tokens_out":90,"duration_ms":7890,"temperature":1.0,"reasoning_tokens":940,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T09:09:01.190240+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Construct a pair of embeddings whose cross-attention maps are nearly identical yet whose score residual still has a large normal component on the true data manifold; if such pairs routinely appear, the attention-geometry bridge fails and AutoAnchor’s stability claims collapse.","supporting_citations":[],"review_version":1}