{"id":"72514ffb-a06b-4665-a94a-3a84afd9f3ff","arxiv_id":"2502.07829","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"This survey provides a taxonomy of preference alignment methods for diffusion models in image generation and editing, covering optimization techniques, applications, and open challenges.","lead":"This paper is a survey, not a new experiment; it organizes and compares existing methods that use human preferences to fine-tune diffusion models for image generation and editing. It is worth reading if you want a structured map of RLHF, DPO, and related techniques in this fast-moving area.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3 mischaracterizes D3PO as using a pre-trained Q function; this factual error undermines the survey's systematic-organization claim and warrants correction.","rationale":"The central claim consists of two parts: (1) that this is the first survey centered on preference alignment with DMs, and (2) that the survey provides a systematic, comprehensive organization of the field. The 'first' claim is a novelty assertion that, even if challenged by prior tutorials (e.g., Uehara et al. 2024), does not by itself determine the survey's usefulness. The 'systematic organization' claim, however, rests entirely on the accuracy of the taxonomy and the method summaries in Table 1 and Sections 3.2-3.4. The D3PO description is a concrete, checkable instance where the summary is likely false: it attributes to D3PO the use of a pre-trained action-value function Q, which contradicts the very design of D3PO as a reward-model-free DPO adaptation. This is exactly the kind of error that a reader would rely on the survey to avoid; if it appears in a prominent baseline, the survey cannot be considered reliable as a systematic reference. The reader's conditional verdict is the right disposition: the paper has potential value, but it requires a correction pass over the method summaries and a transparent selection protocol before it can be accepted as comprehensive. My concern thus reinforces the reader's weakest-assumption analysis rather than moving the verdict.","tokens_in":12230,"tokens_out":6702,"duration_ms":57265,"concrete_test":"Open the D3PO paper (Yang et al., CVPR 2024, 'Using human feedback to fine-tune diffusion models without any reward model') and verify whether it uses a pre-trained action-value function Q. Specifically, search the paper's text for 'Q function', 'action-value', and 'critic'. If D3PO does not use a pre-trained Q model, then Section 3.3's description is false; the authors should correct that summary and audit all other Table 1 entries against their source papers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The survey's value hinges on its summaries being faithful to the cited methods. Section 3.3 states of D3PO (Yang et al., CVPR 2024): 'Using a pre-trained model to represent the action value function Q, the DPO framework is extended to allow direct parameter updates at each denoising step.' In the actual D3PO paper, D3PO adapts DPO to the denoising process by treating it as a multi-step MDP and using pairwise preference data; it does not use a pre-trained action-value function Q. This is not a minor wording slip: D3PO is a core baseline in the survey's Offline Methods block, and the entire point of D3PO is to avoid any reward model or critic. The error appears in the main taxonomy table entry and in the prose. If this entry is inaccurate, readers cannot trust the other ~30 method descriptions in Table 1. Since the central claim is that the survey provides a 'comprehensive and systematic review,' a demonstrably false description of a canonical method is a load-bearing defect, not a stylistic one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a survey of preference alignment methods for diffusion models (DMs), focused on image generation and editing. It proposes a taxonomy based on sampling (online/offline), training strategy (RLHF, DPO, others), reward feedback (human vs. AI), and modality (textual/visual). The survey reviews representative methods in Sections 3.2–3.4, lists them in Table 1, discusses applications in medical imaging, robotics, and autonomous driving in Section 4, and closes with challenges and future directions. The authors claim this is the first survey specifically centered on preference alignment with DMs for image generation and editing.","tokens_in":12432,"tokens_out":5590,"duration_ms":49473,"significance":"If accurate, this survey would fill a genuine gap in the literature by organizing a rapidly growing body of work at the intersection of diffusion models and preference-based optimization. The proposed taxonomy is reasonable, the coverage of methods is broad (approximately 30 methods in Table 1), and the paper identifies relevant challenges such as computational cost, data scarcity, and multi-objective alignment. The survey also has practical value for newcomers entering this area. However, the manuscript's reliability as a reference depends critically on the faithfulness of each method summary; a demonstrable mischaracterization of a canonical method (D3PO) significantly weakens the survey's credibility. The paper also includes many works in Section 4 that do not obviously involve preference alignment, which undermines the claimed application coverage. The work is not a technical research contribution but a survey, so the core assessment is about accuracy and organization.","major_comments":[{"comment":"The description of D3PO is factually incorrect. The manuscript states: \"Using a pre-trained model to represent the action value function Q, the DPO framework is extended to allow direct parameter updates at each denoising step.\" D3PO (Yang et al., CVPR 2024) does not use any pre-trained action-value function Q; its central design is to extend DPO to the denoising process by treating it as a multi-step MDP and applying a step-wise DPO objective, thereby avoiding any reward model or critic. This error is load-bearing because the survey's value derives from accurate method summaries, and D3PO is presented as a core offline method in Table 1. The authors should correct the D3PO summary in Section 3.3 and audit the corresponding row in Table 1, and in light of this error, verify every other method description against its cited source.","section":"Section 3.3, DPO and its variants; Table 1"},{"comment":"Many works described in Section 4 are not preference-aligned diffusion models, despite the abstract's claim that the paper \"thoroughly explore[s] the applications of aligning preferences with DMs in autonomous driving, medical imaging, robotics, and more.\" For instance, Section 4.1 presents MedSegDiff and MedSegDiff-V2 as medical image segmentation methods with no mention of preference alignment or human feedback; Section 4.2 includes Janner et al. (2022) for trajectory planning, a work that does not involve preference alignment. If these are intended as examples of diffusion models used in the respective domains, the section's scope should be explicitly broadened; if they are intended as applications of preference-aligned DMs, the descriptions should clearly state how preference alignment is used. As written, the application coverage is not aligned with the paper's stated scope, and this is a load-bearing issue for the survey's completeness and coherence.","section":"Section 4, Applications"}],"minor_comments":[{"comment":"The text states that DMs can be described as a \"Markov Chain Process (MDP)\"; the correct term is \"Markov Decision Process.\" The acronym MDP is standard and should be expanded correctly.","section":"Section 3.2, RLHF"},{"comment":"The DDIM backward-step equation is typeset in a manner that makes it hard to parse (e.g., \"xt−1 = √ ¯αt−1 xt − √1 − αtϵθ(xt, t)√αt + p 1 − αt−1ϵθ(xt, t)\"). The formula appears to have misplaced parentheses and missing division; please reformat it clearly, and consider referencing the original paper for the exact expression.","section":"Section 2.1, DDIM equation"},{"comment":"Many reference entries are incomplete: e.g., \"Guan and et al.\", \"Gao and et al.\", \"Schulman and et al.\", \"Prabhudesai and et al.\", and \"Yang and et al.\" lack the first author's full name. This deviates from standard citation style and makes verification difficult. Full author lists (or at least first author names) should be provided.","section":"References"},{"comment":"The submitted version of Table 1 does not visibly separate the column headers from the data rows, making it difficult to determine which method corresponds to which combination of training strategy, task, reward feedback, and modality. Please reformat the table so that column headers are clearly aligned with their entries.","section":"Table 1"},{"comment":"The future-directions discussion is mostly qualitative and repeats common themes (efficiency, data, multimodality, safety). It would benefit from concrete pointers to the methods reviewed earlier (e.g., which of the surveyed approaches already address these issues) and from references to recent work on each direction.","section":"Section 5, Challenges"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is a survey, so the accuracy of method summaries is the primary scientific criterion. The D3PO error is significant and should be corrected before publication; I also recommend that the authors systematically re-check all entries in Table 1 against the cited papers, since one error in a canonical method reduces confidence in the rest. Section 4's scope mismatch is a second substantive issue: either the section should be explicitly framed as 'diffusion applications that can benefit from preference alignment,' or the descriptions should be revised to indicate precisely where preference alignment is used. The reference list needs a thorough cleanup. The 'first survey' claim is plausible but should be phrased with more care, since the authors themselves cite reviews on RL fine-tuning of diffusion models; the novelty lies in the specific focus on image generation and editing with preference alignment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this survey is a genuinely useful organizing map of preference alignment for diffusion models, but it ships with a factual error on D3PO that needs fixing before it can be trusted as a reference.\n\nWhat's actually new: the taxonomy (sampling, training strategy, reward feedback, modality) is a sensible way to slice the field, and Table 1 is a compact comparison of ~40 methods that will save people time. The application sweep (medical, robotics, driving) is broader than most similar reviews, and the background equations on DDPM/DDIM/PPO are standard and correctly stated.\n\nWhat's soft: the D3PO description in Section 3.3 is wrong. The actual D3PO paper explicitly avoids any reward model or critic; it reframes denoising as a multi-step MDP and applies DPO per step with human pairwise preferences. The sentence about a 'pre-trained model to represent the action value function Q' is not what D3PO does, and it contradicts the same paragraph's claim that D3PO eliminates the need for a reward model. Since D3PO is a canonical offline method in Table 1, this is a substantive accuracy problem, not a typo. The reader's stress test is correct.\n\nThere are also citation-format problems: many references use 'and et al.' without the first author name (e.g., 'Guan and et al., 2024', 'Schulman and et al., 2015'), which is sloppy and makes verification harder. And the survey has no stated search/inclusion protocol, so the 'comprehensive' claim is hard to assess. The 'first survey' claim is mildly overstated given earlier RL-for-diffusion surveys, though the narrower scope on preference alignment is defensible.\n\nCredit where due: most of the method summaries I spot-checked (DDPO, DPOK, Diffusion-DPO, SPIN-Diffusion) are accurate, and the framework figure is helpful.\n\nBottom line: this is a useful entry point for newcomers, but the D3PO error and citation issues mean it shouldn't be circulated as-is. It deserves a serious referee and a conditional accept with mandated corrections. I wouldn't cite it in its current form; after fixes, I'd consider it.","headline":"Useful taxonomy and a broad method sweep, but the D3PO mischaracterization and sloppy citations keep it from being a reliable reference without revision.","tokens_in":12929,"tokens_out":2765,"would_cite":false,"duration_ms":22550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey claims to be the first comprehensive map of preference alignment for diffusion models, organizing methods by sampling, training strategy, reward feedback, and modality.","keywords":["preference alignment","diffusion models","image generation","image editing","reinforcement learning","RLHF","DPO","taxonomy"],"falsifier":"A literature search for a previously published survey whose stated scope is preference alignment with diffusion models for image generation and editing would settle the priority claim: finding one falsifies it. To test the taxonomy, a reader could re-read each cited paper and check the Table 1 cells (online or offline, RLHF or DPO or other, human or AI feedback, text or visual or multimodal); systematic mismatches would falsify the organizing claim.","tokens_in":12059,"feed_emoji":"🎨","tokens_out":10316,"duration_ms":80045,"temperature":0.7,"pith_summary":"The paper sets out to fill a gap: surveys of diffusion-based generation and editing exist, but none, in the authors' assessment, focuses specifically on preference alignment, the family of techniques that fine-tune diffusion models using human or AI judgments about what makes an output good. To make that family tractable, the paper casts all such methods as reinforcement-learning problems and sorts them into a taxonomy with four axes: sampling, training strategy, reward feedback, and modality. It reviews reinforcement learning with human feedback (RLHF), direct preference optimization (DPO) and its variants, and less common alternatives such as GFlowNets (generative flow networks, a class of reward-proportional samplers) and self-play, then maps applications in medical imaging, robotics, autonomous driving, and adjacent domains. It closes by naming computational cost, preference-data collection, multi-objective consistency, and reward subjectivity as the core challenges. A sympathetic reader would take the paper's contribution to be organizational: a structured entry point into a fast-moving intersection.","feed_headline":"Four-axis taxonomy maps preference alignment for diffusion models","feed_subtitle":"RLHF, DPO, and alternatives are sorted into one framework, with applications from medical imaging to autonomous driving.","key_machinery":"The load-bearing machine is a taxonomy built on two moves. First, preference alignment is formulated as an RL problem in which the diffusion model is the policy, human or AI judgments supply a reward, and the denoising process is the environment. Second, every method is classified along four axes: sampling (online vs offline), training strategy (RLHF, DPO, or others, including GFlowNets, a class of reward-proportional samplers, and self-play), reward feedback (human vs AI annotation), and modality (textual, visual, or multimodal). The taxonomy is materialized in Table 1, where each surveyed method becomes a row and each axis a column, exposing which combinations of choices have been tried and which remain open.","core_discovery":"The central claim is organizational: the many methods that fine-tune diffusion models with human or AI preferences share an RL-style skeleton—a policy, a reward, an action space, and an environment—and each method can be located by four design axes. The paper asserts that no prior survey has centered on this intersection, so its taxonomy gives the first systematic way to compare methods. If the organization holds, the field becomes easier to navigate for newcomers and easier to mine for open problems by experienced researchers.","pith_inferences":["The four-axis taxonomy doubles as a design checklist: a new method is fully specified once it chooses online or offline sampling, RL or DPO or another training loop, human or AI feedback, and a text, visual, or multimodal target.","The survey does not quantitatively compare methods, so a natural next step is a shared benchmark that evaluates methods within each taxonomy cell to test whether the axes predict performance.","The same RL formulation should extend to video and 3D diffusion models, because the reward and preference signals transfer even though the denoising environment changes."],"forward_implications":["A newcomer can locate any given method in the taxonomy via Table 1, which splits methods by online versus offline sampling and by RLHF, DPO, or other training strategies.","The survey implies that pairwise preference data can replace a learned reward in many diffusion tasks, since DPO bypasses reward modeling and RL updates altogether.","The application chapters show the same alignment machinery moving across medical imaging, robotics, autonomous driving, biology, games, and content creation, suggesting a shared toolkit rather than domain-specific algorithms.","The challenges section turns computational cost, preference-data collection, multi-objective consistency, and reward subjectivity into concrete research targets.","The future-directions discussion identifies parameter-efficient tuning, multimodal preference interpretation, robust evaluation metrics, and safety and privacy as the next frontiers."],"supporting_citations":[{"why":"Defines denoising diffusion probabilistic models, the base architecture that all surveyed alignment methods fine-tune.","marker":"Ho et al., 2020"},{"why":"Introduces deep RL from human preferences, the foundation of the RLHF pipeline the survey reviews.","marker":"Christiano et al., 2017"},{"why":"Supplies the Proximal Policy Optimization algorithm used as the stable RL optimizer in RLHF fine-tuning.","marker":"Schulman et al., 2017"},{"why":"DDPO recasts diffusion denoising as an MDP and is the survey's primary online RLHF example.","marker":"Black et al., 2023"},{"why":"DPOK adds KL regularization to the reward objective and is the survey's second canonical RLHF method.","marker":"Fan et al., 2024"},{"why":"Introduces direct preference optimization, the main RL-free alternative that the survey contrasts with RLHF.","marker":"Rafailov et al., 2024"},{"why":"Diffusion-DPO adapts DPO to text-to-image diffusion models and anchors the survey's DPO-variant category.","marker":"Wallace et al., 2024"},{"why":"Human Preference Score trains a reward predictor on human choices, an early instance of the human-feedback data axis.","marker":"Wu et al., 2023"}],"fun_headline_variants":["First survey maps diffusion preference alignment","Four axes sort diffusion preference alignment methods","RLHF to DPO: one framework for diffusion tuning","Diffusion preference alignment: a unified taxonomy","Navigating diffusion preference alignment with four axes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's usefulness depends on each summarized method being described faithfully and assigned to the correct cell of its taxonomy.","fun_headline_variants_meta":{"raw":{"variants":["First survey maps diffusion preference alignment","Four axes sort diffusion preference alignment methods","RLHF to DPO: one framework for diffusion tuning","Diffusion preference alignment: a unified taxonomy","Navigating diffusion preference alignment with four axes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1784,"prompt_tokens":795,"completion_tokens":989,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":411,"completion_tokens_details":{"reasoning_tokens":923}},"tokens_in":411,"tokens_out":989,"duration_ms":8092,"temperature":1.0,"reasoning_tokens":923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:01:08.012622+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A literature search for a previously published survey whose stated scope is preference alignment with diffusion models for image generation and editing would settle the priority claim: finding one falsifies it. To test the taxonomy, a reader could re-read each cited paper and check the Table 1 cells (online or offline, RLHF or DPO or other, human or AI feedback, text or visual or multimodal); systematic mismatches would falsify the organizing claim.","supporting_citations":[{"cited_title":"Deep reinforcement learning from human preferences","cited_arxiv_id":null,"evidence_quote":"Introduces deep RL from human preferences, the foundation of the RLHF pipeline the survey reviews."},{"cited_title":"Proximal policy optimization algorithms","cited_arxiv_id":null,"evidence_quote":"Supplies the Proximal Policy Optimization algorithm used as the stable RL optimizer in RLHF fine-tuning."},{"cited_title":"Training diffusion models with reinforcement learning","cited_arxiv_id":null,"evidence_quote":"DDPO recasts diffusion denoising as an MDP and is the survey's primary online RLHF example."},{"cited_title":"Direct preference optimization: Your language model is secretly a reward model","cited_arxiv_id":null,"evidence_quote":"Introduces direct preference optimization, the main RL-free alternative that the survey contrasts with RLHF."},{"cited_title":"Diffusion model alignment using direct preference optimization","cited_arxiv_id":null,"evidence_quote":"Diffusion-DPO adapts DPO to text-to-image diffusion models and anchors the survey's DPO-variant category."}],"review_version":1}