{"id":"a6e22b0d-8024-40d0-8365-938c70d86bd5","arxiv_id":"2607.03397","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Joint concept-graph alignment of Stable Diffusion’s text encoder and denoiser cuts fairness discrepancy ~30%, incoherent outputs 88%, and improves adversarial unlearning robustness.","lead":"CO-ALIGN jointly rewires concept graphs inside a text-to-image model’s text encoder and denoiser so demographic stereotypes weaken without wrecking image coherence. The same edit also makes later concept-unlearning harder to bypass with nearby adversarial prompts.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The causal fidelity of the extracted concept graphs remains the load-bearing soft spot; output gains alone do not prove the edges drive demography.","rationale":"The reader correctly isolates the weakest assumption: that the particular graph constructions in §4–5 encode a causal ontology whose edge registration produces the reported fairness/coherence gains. The paper’s evidence is strong on the engineering claim (joint alignment beats single-component SOTA on the DiffLens protocol and helps unlearning) but only indirect on the mechanistic claim. No contradiction or circularity appears; the concern is simply that the causal link is untested. Because the reader already flags this as the reason for CONDITIONAL rather than ACCEPT, and because the concrete edge-perturbation test would settle it without requiring a broader re-evaluation of scope or classifiers, the appropriate stress-test outcome is to leave the verdict unchanged. The same concern is not fatal: the algorithms, hyper-parameters, and sequential TE→denoiser procedure are concrete enough that a competent group can re-implement and run the proposed causal check. Novelty and empirical utility remain intact inside the reported setting.","tokens_in":19934,"tokens_out":614,"duration_ms":40905,"concrete_test":"After full CO-ALIGN, freeze all weights and surgically re-perturb only the supervised target-row edges of A_U (or the corresponding concept-neuron fingerprints) back toward the original biased topology while leaving the text encoder and all other parameters untouched; regenerate the 500-image occupation suite. If FD reverts toward the original/base values while CLIP-T and incoherence stay near the aligned levels, the edges are causally load-bearing; if FD remains low, the fairness gains are not primarily mediated by the registered concept-graph edges.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that A_TE (Eq. 4: cosine of mean-pooled CLIP embeddings) and A_U (Eqs. 5–8: average cosine of masked K/Q/V concept-neuron gradient fingerprints) are faithful proxies for the causal internal concept ontology, so that registering equal target-to-attribute edge weights (Eq. 9, s*=1.0) via the ranking loss (Eqs. 10–12) produces balanced generations. The paper shows that joint registration improves FD/FID/CLIP-I and reduces incoherence relative to single-component baselines (Table 2, Fig. 2b), and that sequential TE-then-denoiser alignment restores coherence (Fig. 4). These are correlational output statistics. No intervention demonstrates that the demographic distribution is controlled by the registered edges themselves rather than by other pathways (e.g., residual stream geometry, non-CA layers, or the preservation/anchor losses). If the graphs are only loosely coupled to the generative pathway that actually samples gender/race/age, the method could still succeed for incidental reasons while the ontology-alignment story would not hold.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes CO-ALIGN, a bias-mitigation method for text-to-image diffusion models that extracts concept graphs from the text encoder (pairwise cosine of mean-pooled embeddings, Eq. 4) and the denoiser (average cosine of masked K/Q/V concept-neuron gradient fingerprints, Eqs. 5–8), then registers a hand-specified target topology of equal proximity between an underspecified concept and demographic anchors (Eq. 9) via a differentiable ranking loss (Eqs. 10–12), applied sequentially to both components with anchor and preservation regularizers. On Stable Diffusion v1.5 under the DiffLens four-occupation protocol, joint alignment is reported to improve fairness discrepancy (e.g., gender FD 0.032), FID, CLIP-I, and to cut semantic incoherence relative to single-component baselines (Table 2, Fig. 2b), with a secondary application in which pre-aligning adversarial bypass concepts improves post-hoc unlearning robustness (Table 3) via an observed neighbourhood-pulling effect (Fig. 5).","tokens_in":20250,"tokens_out":1707,"duration_ms":56949,"significance":"If the empirical claims hold under broader scrutiny, the work is significant for two reasons. First, it directly targets a documented failure mode of current debiasing—fairness gains that trade off into incoherent generations when only the text encoder or only the denoiser is edited—and places joint ontology-style alignment on a better fairness–coherence Pareto front than several strong baselines. Second, the neighbourhood-propagation observation and its use as a pre-alignment step for UCE/ESD/SAFREE is a concrete, reusable idea for strengthening concept unlearning without changing the unlearning procedure. The manuscript also supplies extraction/alignment algorithms, a reproducibility appendix with hyperparameters and wall-clock costs, and multi-metric reporting (FD, FID, CLIP-I/T, VLM incoherence), which strengthens the contribution relative to purely inference-time steering papers.","major_comments":[{"comment":"§4–5, Eqs. (4)–(9) and the central ontology claim: the load-bearing premise is that A_TE and A_U are faithful proxies for the causal internal concept ontology, so that equalizing target–attribute edge weights produces demographic balance. Table 2 and Figs. 2b/4 show that joint registration improves FD/FID/CLIP-I and restores coherence relative to single-component baselines, but these are output correlations. There is no intervention or ablation that isolates the registered edges (e.g., randomizing off-target graph rows, matching FD with a non-graph control that only applies L_pres/L_anchor, or intervening on non-CA pathways) from residual-stream geometry, non-CA layers, or the regularizers. Without such evidence, the method’s gains may be real while the “concept-ontology alignment” causal story remains under-supported; please add at least one edge-level or pathway-control experiment.","section":"§4–5, Eqs. 4–9"},{"comment":"Experimental setup / Table 2: all primary debiasing claims rest on four occupations (nurse, doctor, firefighter, receptionist) and three axes under the DiffLens protocol, with FD/FID/CLIP averaged over that set. That protocol is a reasonable starting point, but the abstract’s SOTA claims (30% fairness, ΔFID=11.4, 2.8% fidelity, 88% incoherence reduction) are stated generally. The manuscript needs either a broader occupation/prompt suite (including non-face and multi-attribute prompts) or a clearly scoped claim limited to this protocol, plus per-occupation breakdowns and variance over seeds so that the averages cannot be driven by one easy profession.","section":"§6, Table 2"},{"comment":"Abstract and §5 claim effectiveness “across three paradigms: text-encoders, denoisers and joint,” yet Table 2 reports only joint CO-ALIGN against external baselines. Fig. 4 qualitatively shows TE-only incoherence repaired by subsequent denoiser alignment, and Fig. 2b places other methods, but the paper lacks a controlled within-framework ablation (same loss, same anchors, TE-only vs UNet-only vs joint) with the full metric suite. That ablation is necessary to substantiate the joint-alignment hypothesis as the reason for the Pareto improvement rather than the particular ranking objective or LoRA/UNet fine-tuning recipe.","section":"Abstract; §5–6; Fig. 4"},{"comment":"§5, Eq. (9) and free design choices: the target topology fixes s*=1.0 (equal proximity to all anchors) and a small supervised row set S={c*}, with free parameters τ, δ, ξ, LoRA rank/scale, and sequential TE-then-denoiser order. Fairness is then measured by FD against a uniform demographic target. The paper should report sensitivity of FD/FID/incoherence to s*, to the choice of anchors, and to TE↔denoiser order; otherwise it is unclear whether success depends on a narrow hyperparameter island or on the graph-registration idea itself. A short sensitivity table would make the central claim much more robust.","section":"§5, Eq. 9; Appendix E"}],"minor_comments":[{"comment":"Several abstract/intro percentages (30%, ΔFID=11.4, 2.8%, 88%) should be tied explicitly to the exact baseline and metric column in Table 2 / Fig. 2b so readers can recompute them.","section":"Abstract; §6"},{"comment":"Figure 1 and related captions contain OCR-like artifacts (“T xt Enc”, “corr ection”, “CG G T”) and the abstract has a missing space (“intosemantically”); clean for camera-ready.","section":"Fig. 1; Abstract"},{"comment":"Table 1 marks CO-ALIGN as not training-free and with no added inference cost, which is fair, but wall-clock (TE ~5 min, UNet ~3 h on A100) belongs in the main comparison discussion so efficiency claims are not only in Appendix E.","section":"Table 1; Appendix E"},{"comment":"Incoherence and nudity metrics rely on LLaVA zero-shot prompts (Appendix F). Please report inter-rater agreement against a human sample or a second VLM, and release the exact matching rules used for “None” / “yes”.","section":"Appendix F; Fig. 2b; Table 3"},{"comment":"Table 3: CO-ALIGN alone raises average flagged nudity (60.4% vs 48.4%) before unlearning; discuss this side effect briefly so the pre-alignment step is not read as universally safety-improving in isolation.","section":"§7, Table 3"},{"comment":"Notation: A_TE / A_U / A_GT and G_TE / G_U are clear, but “concept neuron fingerprint” vs TRUST’s unlearning use could be one sentence sharper to avoid conflating methods.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The empirical Pareto result against single-component debiasing is the publishable core if the joint-vs-single ablation and some edge-control evidence are added; without them the ontology framing is stronger than the causal support. Scope is narrow (SD1.5, four occupations) for a general “efficient bias mitigation” title—worth watching whether the revision broadens evaluation or narrows claims. No integrity red flags beyond ordinary over-claiming in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: single-component debiasing on SD1.5 trades fairness for incoherent generations, and jointly registering concept graphs on both the text encoder and the UNet cross-attention layers largely removes that trade-off. Table 2 and Fig. 2(b) put CO-ALIGN on the Pareto front against Asyrp, H-Distribution, Latent Directions, Finetuning, DiffLens, and UCE on the four-occupation DiffLens protocol (gender FD 0.032, race 0.043, ΔFID ~11 below base, CLIP-I up ~2.8%, incoherence down ~88%). The sequential TE-then-denoiser repair in Fig. 4 is the cleanest illustration of the diagnosis.\n\nWhat is actually new is the dual extraction (cosine of mean-pooled TE embeddings; average cosine of masked K/Q/V concept-neuron gradient fingerprints) plus the differentiable ranking registration to a hand-specified equal-proximity target, with static anchors and a preservation term. The neighborhood-pulling effect and its use as a pre-alignment for UCE/ESD/SAFREE (Table 3: average nudity rates dropping to 0.8–10%) is a genuine extra contribution, not a re-label of existing unlearning.\n\nThe soft spots are real but proportionate. Evaluation is narrow: four occupations, three FairFace axes, LLaVA oracles, no uncertainty, no code/checkpoints. The free parameters (τ, δ, ξ, s*=1.0, LoRA rank) are standard and not hidden. The stress-test concern is fair: the paper never shows that the registered edges themselves are the causal pathway for demography rather than residual geometry or the anchor/preservation losses. That keeps the ontology story correlational. It does not erase the output gains or make the method circular; the target topology is not fitted to FD.\n\nMath and algorithms are clear enough to re-implement with non-trivial engineering. Citations cover the right TE-only and denoiser-only lines. This is for people working on T2I fairness and post-hoc safety who need a practical joint edit, not a training-data solution. I would send it to peer review; it deserves referee time and broader validation, not a desk reject.","headline":"Joint TE+denoiser concept-graph registration is a clean, usable fix for the fairness–coherence trade-off on SD1.5, with a real unlearning side-effect; the ontology story is still correlational.","tokens_in":20875,"tokens_out":570,"would_cite":true,"duration_ms":5213,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Bias in text-to-image models is fixed by jointly aligning concept graphs in the text encoder and denoiser, not by editing one side alone.","keywords":["text-to-image diffusion","bias mitigation","concept graphs","text encoder","denoiser","concept ontology alignment","concept unlearning","Stable Diffusion"],"falsifier":"Run the same joint-alignment pipeline on Stable Diffusion v1.5 but replace the concept-graph registration loss with a matched control that randomizes edge targets or freezes one component; if fairness, coherence, and unlearning gains disappear while image quality stays intact, the claim that graph alignment itself is the causal mechanism is falsified.","tokens_in":20759,"feed_emoji":"⚖️","tokens_out":650,"duration_ms":34734,"temperature":0.7,"pith_summary":"Text-to-image diffusion models inherit demographic stereotypes from their training data and then amplify them. Methods that only rewrite the text encoder or only edit the denoiser improve fairness numbers but often produce images that no longer depict the requested concept, because the two components no longer share a coherent geometry. This paper argues that the real problem lives in the asymmetric relationships among concepts inside both components: a prompt like “nurse” is wired more tightly to female attributes than to male ones. CO-ALIGN extracts a concept graph from each side—embedding similarities for the text encoder and shared concept-neuron fingerprints for the denoiser—and registers both graphs toward a balanced target topology. Joint alignment restores demographic balance while keeping generations coherent and photorealistic. The same pre-alignment also pulls adversarial bypass concepts into the neighbourhood of an unlearning target, making existing concept-erasure techniques far harder to circumvent.","feed_headline":"Joint concept-graph alignment cuts T2I bias 30% without collapse","feed_subtitle":"Editing text encoder and denoiser together restores fairness, image quality, and unlearning robustness","key_machinery":"CO-ALIGN: extract a text-encoder concept graph from cosine similarities of mean-pooled embeddings and a denoiser concept graph from average cosine of masked concept-neuron gradient fingerprints over cross-attention K/Q/V weights, then align both adjacency matrices to a target graph that equalizes the target concept’s proximity to its attribute anchors via a differentiable ranking loss plus static-anchor and preservation terms.","core_discovery":"Effective debiasing of text-to-image diffusion requires joint registration of concept graphs extracted from the text encoder and the denoiser toward a common target topology; single-component edits break the geometric coupling between the two and trade residual bias for semantically incoherent outputs, whereas CO-ALIGN simultaneously improves fairness, image quality, fidelity, and coherence, and further strengthens post-hoc unlearning against adversarial prompts.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Joint concept-graph alignment cuts T2I bias 30% with 88% fewer collapses","CO-ALIGN jointly registers encoder-denoiser graphs for fairer T2I outputs","Aligning text and denoiser ontologies boosts fairness 30% plus fidelity","Single-module edits fail: joint concept graphs restore T2I fairness and quality","Concept-graph co-alignment debiases diffusion without semantic collapse"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The premise that these particular graphs of embedding similarities and shared concept-neuron fingerprints truly capture the causal internal concept ontology whose edge rebalancing will produce fairer generations.","fun_headline_variants_meta":{"raw":{"variants":["Joint concept-graph alignment cuts T2I bias 30% with 88% fewer collapses","CO-ALIGN jointly registers encoder-denoiser graphs for fairer T2I outputs","Aligning text and denoiser ontologies boosts fairness 30% plus fidelity","Single-module edits fail: joint concept graphs restore T2I fairness and quality","Concept-graph co-alignment debiases diffusion without semantic collapse"]},"model":"grok-4.5","effort":"low","cost_usd":0.005306,"raw_usage":{"total_tokens":1449,"prompt_tokens":751,"num_sources_used":0,"completion_tokens":108,"cost_in_usd_ticks":53060000,"prompt_tokens_details":{"text_tokens":751,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":590,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":751,"tokens_out":108,"duration_ms":4647,"temperature":1.0,"reasoning_tokens":590,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T02:47:12.787374+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same joint-alignment pipeline on Stable Diffusion v1.5 but replace the concept-graph registration loss with a matched control that randomizes edge targets or freezes one component; if fairness, coherence, and unlearning gains disappear while image quality stays intact, the claim that graph alignment itself is the causal mechanism is falsified.","supporting_citations":[],"review_version":1}