Pith. sign in

REVIEW 5 major objections 5 minor 9 cited by

A 10-minute fine-tuning run that rewards the difference between 'Realistic photo' and 'CG Render' scores raises FLUX's human-rated realism excellent rate from 8.2% to 38.9% without new data or reward-model retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 22:51 UTC pith:IDXTOZHB

load-bearing objection A genuinely useful training trick (ground-truth noise injection) makes early-timestep RL feasible, but the headline 3x human-eval gain rests on a human study that needs more rigor before I'd fully trust it. the 5 major comments →

arxiv 2509.06942 v3 pith:IDXTOZHB submitted 2025-09-08 cs.AI cs.LG

Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference

Xiangwei Shen , Zhimin Li , Zhantao Yang , Shiyi Zhang , Yingfang Zhang , Donghao Li , Chunyu Wang , Qinglin Lu
show 1 more author
Yansong Tang
This is my paper
classification cs.AI cs.LG
keywords text-to-image generationonline reinforcement learningreward hackingsemantic-relative rewarddiffusion trajectory optimizationphotorealismhuman preference alignmentflow-matching models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that text-to-image diffusion models can be aligned to fine-grained human preferences, such as photorealism, by optimizing over the full denoising trajectory rather than only its final steps. It introduces Direct-Align, which injects known Gaussian noise into a clean training image so the image can be recovered in one closed-form step from any timestep, making early-timestep optimization cheap and stable. It then redefines the reward as the difference between a reward model's scores under a positive control prompt ('Realistic photo') and a negative one ('CG Render'), a semantic-relative reward that cancels the reward model's general biases and can be steered online by prompt augmentation. On FLUX.1.dev, the method lifts human 'excellent' scores from 8.2% to 38.9% for realism and 9.8% to 40.5% for aesthetics after about 10 minutes of training, with no additional data and no reward-model retraining. If correct, this makes online reinforcement learning with a generic preference reward a practical route to substantially better human-perceived quality in a large open-weight model, and the same recipe may transfer to other diffusion and flow-matching models.

Core claim

The central claim is that the two obstacles to direct reward backpropagation in diffusion models—expensive multistep denoising and reward hacking from late-stage-only optimization—can both be removed by treating the diffusion trajectory as a reversible interpolation. Direct-Align uses the forward-construction identity x_t = α_t x_0 + σ_t ε, with ε the injected ground-truth Gaussian noise, to place a clean image at any timestep and then recover it exactly as x_0 = (x_t − σ_t ε)/α_t, so reward gradients reach early denoising steps without iterative sampling or gradient explosion. SRPO then computes the reward as the difference between HPSv2.1 scores under opposing prompt conditions, r = f_img(

What carries the argument

The load-bearing identity is the diffusion interpolation equation x_t = α_t x_0 + σ_t ε, which states that any noisy state is a weighted sum of a clean image and known Gaussian noise. Direct-Align inverts this analytically to recover the clean image in one step from any timestep, letting reward gradients backpropagate through early, high-noise stages of the trajectory that previous methods had to skip. The second mechanism is the semantic-relative reward r_SRP = f_img(x)·(C_1 − C_2), the difference between reward-model scores under a positive control prompt and a negative control prompt; it defines the optimization direction as the semantic gap between the targeted attribute and an unwanted

Load-bearing premise

The method works only if the gap between the reward model's scores for 'Realistic photo' and 'CG Render' measures realism and little else; if that text-embedding difference also tracks color, texture, or style, the optimizer is chasing a proxy, and the headline gains rest on a ten-annotator panel with no reported inter-rater agreement.

What would settle it

Swap the control pair: run the same 10-minute fine-tune with 'CG Render' as the positive control and 'Realistic photo' as the negative one, then measure human excellent rates. If the semantic difference C1 − C2 carries the effect, judged realism should fall, not rise; if realism still improves regardless of which pole is positive, the claimed mechanism is not the one doing the work. Independently, re-running the human evaluation with a larger annotator pool and reported confidence intervals would settle whether the 30-point jump in excellent rate is a stable effect or an artifact of the small

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Training across the full trajectory, including early high-noise timesteps, is what suppresses reward hacking: ablations restricted to late timesteps show a sharply higher annotated 'hacking rate' and oversaturated artifacts.
  • Because the reward is prompt-conditioned, attribute control becomes an online, prompt-level operation: prepending words such as 'Dark', 'Golden', 'Oil Painting', or 'Concept Art' shifts the fine-tuned model's output distribution along those axes without any reward-model retraining.
  • The same short procedure works across different reward models—CLIP, PickScore, HPSv2.1, and the unimodal Aesthetic Predictor 2.5—with no observed reward hacking, supporting the claim that the relative formulation, not the choice of reward model, carries the effect.
  • The method matches or exceeds prior online-RL baselines (ReFL, DRaFT-LV, DanceGRPO) on human evaluation while using about 75× less GPU time than the GRPO-based baseline.
  • Substituting offline real photographs for online rollouts in the same Direct-Align loop further improves realism, indicating the mechanism doubles as a data-fitting procedure when the reward carries no human-preference signal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The success of the 'Realistic photo' versus 'CG Render' difference suggests the reward model's text encoder stores realism as a recoverable semantic direction; a natural extension is to test whether other attribute axes (sharp vs soft, natural vs saturated, bright vs dim) respond with the same magnitude, or whether only attributes strongly represented in the reward model's training data respond.
  • If the mechanism is real, it should compound with stronger base reward models: pairing the relative-reward trick with a finer-grained human-preference model could push human excellent rates higher still, because the relative formulation filters bias rather than replacing the signal.
  • The paper's own limitation notes that control words outside the reward model's domain ('Renaissance', 'Cyberpunk') respond weakly or produce artifacts; this predicts a testable monotonic relationship between a word's frequency in the reward model's training set and its steering strength.
  • The inversion-based regularization suggests a general 'anti-style' tool: choosing the right negative control prompt could actively suppress known failure modes such as glossiness, oversaturation, or edge artifacts during fine-tuning, independent of which positive attribute is being optimized.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Direct-Align and Semantic Relative Preference Optimization (SRPO) for fine-tuning text-to-image diffusion models (specifically FLUX.1-dev) on differentiable reward models. Direct-Align injects a known Gaussian noise into a clean image to obtain an intermediate diffusion state and then recovers the image using the ground-truth noise combined with a small model-prediction term, allowing reward gradients to be computed at arbitrary timesteps without backpropagating through a multistep sampler. SRPO formulates the reward as a text-conditioned signal: for each sample, it takes the difference between reward scores under a positive control phrase (e.g., "Realistic photo") and a negative control phrase (e.g., "CG Render"), and optionally uses an inversion-based regularization. The authors report that after 10 minutes of training on HPDv2 with HPSv2.1, the model's human-evaluated realism and aesthetic quality increase by over 3x (excellent rate from 8.2% to 38.9% for realism, and 9.8% to 40.5% for aesthetics), while automatic metrics such as Aesthetic, PickScore, ImageReward, HPSv2.1, and GenEval show mixed or flat results.

Significance. If the empirical claims held, the work would be significant: it suggests a practical and extremely efficient way to improve photorealism of a large text-to-image model without new data or reward-model retraining, and it introduces a text-conditioned relative reward that could generalize to other online-RL pipelines. The paper contains extensive ablations (Figs. 7-9), cross-reward generalization experiments (Fig. 6), style-control demonstrations (Fig. 8), and a self-aware limitations section. The training efficiency claim (10 minutes on 32 H20 GPUs, 75x faster than DanceGRPO) is noteworthy. However, the central human-evaluation claim is not currently supported by the evidence: the automatic metrics do not independently corroborate the large human-rated gains, and the human evaluation lacks key statistical safeguards. The methodological novelty of Direct-Align is partly undercut by the fact that Eq. (4) is an algebraic identity using the known injected noise.

major comments (5)
  1. [§3.1, Eqs. (3)-(5)] The claim of "exact single-step recovery" from any timestep is tautological: Eq. (4) is obtained by rearranging Eq. (3), and the noise ε_gt is known because the method itself injects it into a clean image x0 (which was generated without gradients). Consequently, Eq. (5) does not recover an image from noise; it computes a reward on the original clean image plus a model-predicted residual term Δσ_t(ε_θ-ε_gt)/α_t. The gradient therefore flows through a local perturbation of the already-generated image, not through a genuinely generated sample. The paper should clarify what is actually being optimized and why this avoids reward hacking beyond the usual late-timestep issue, otherwise the methodological contribution is overstated.
  2. [§4.2, Table 1, SGP-HPS definition] SGP-HPS is defined as the difference between HPSv2.1 scores with positive and negative control prefixes, which is exactly the objective SRPO maximizes (Eqs. 9-10). Reporting an improvement in SGP-HPS from 0.463 to 0.505 is therefore a measure of optimization success, not an independent evaluation of realism. The paper needs a non-circular realism metric (e.g., a held-out realism classifier, FID to real photographs, or human ratings analysed separately) to support the claim that SRPO improves realism beyond the training objective.
  3. [§4.2, Fig. 4, Table 1] The central "over 3x" claim rests entirely on the human "excellent rate" in Fig. 4 and Table 1. The evaluation protocol reports no inter-annotator agreement (e.g., Cohen's kappa or Krippendorff's alpha), no confidence intervals, no statistical test, and no explicit details on blinding or order randomization. The paper itself notes (Sec. 2) that HPDv2 and ImageReward have only 59.7% and 65.7% inter-annotator agreement. With a baseline excellent rate of 8.2%, a relatively small number of rating shifts could produce the observed 38.9% rate, especially if annotators were not blinded to the training condition. The authors should report per-prompt confidence intervals, agreement statistics, and a significance test against the baseline.
  4. [§3.2, Eqs. (9)-(10)] The load-bearing assumption of SRPO is that the embedding difference C1-C2 for "Realistic photo" versus "CG Render" isolates the realism attribute and cancels general reward biases. This assumption is not tested. The paper does not analyze the semantic direction spanned by this text-embedding difference, nor does it verify that the reward difference is dominated by realism rather than by other correlated dimensions (e.g., color grade, texture complexity, style, or artifact presence). The Limitations section itself admits that control tokens can be outside the reward model's domain and that "effects of some control texts may not align with the intended RL direction." The authors should provide evidence, for example by showing that the SGP reward difference correlates with human realism ratings on a held-out set, or by ablating other control-word pairs.
  5. [§4.3, Table 1] Automatic metrics are mixed and do not independently support the human-evaluation claim. In Table 1, HPSv2.1 is unchanged (0.289 vs. 0.289), ImageReward improves only marginally (1.115 vs. 1.118), GenEval decreases (0.678 vs. 0.665), and DeQA slightly decreases (4.292 vs. 4.275). Only Aesthetic and PickScore improve. The paper should explicitly discuss this discrepancy and provide an explanation (e.g., that the human eval targets realism which HPS/GenEval do not measure). Without independent automatic corroboration or stronger human statistics, the 3x claim is not convincing.
minor comments (5)
  1. [Eq. (6)] The reward aggregation formula is garbled: "rk−n k" should presumably be a sum from k-n to k. Please fix the notation and define λ(t) explicitly.
  2. [§4.1] Training hyperparameters (learning rate, number of optimization steps, noise injection schedule, the exact values of Δσ and λ(t)) are not reported. This limits reproducibility.
  3. [Fig. 9A] The comparison with FLUX.1.Krea is unclear because the bar chart labels appear inconsistent (77.7% vs 99.3%). Clarify what the percentages represent and whether they are from the same 500-prompt evaluation set.
  4. [§4.4] When comparing reward models, the paper states that PickScore converges faster than HPS, but no training curves or quantitative convergence data are shown. Adding this would strengthen the claim.
  5. [Section 2] The related-work discussion would benefit from a clearer distinction between direct-backpropagation methods and policy-gradient methods; currently the terminology "Online-RL" is used for both, which may confuse readers.

Circularity Check

2 steps flagged

SGP-HPS is defined as the exact SRPO reward (positive-minus-negative control prompt difference), and Direct-Align's one-step 'recovery' is the algebraic inverse of its own noise injection; the headline 3x human-eval claim still rests on independent human ratings.

specific steps
  1. fitted input called prediction [Sec. 3.2 Eqs. (9)-(10) and Sec. 4.2 'Automatic metrics' / Table 1]
    "Furthermore, we introduce SGP-HPS, which quantifies the difference between score extracted by HPSv2.1 from prompts prefixed with “Realistic photo” (C1) and “CG Render” (C2) using HPSv2.1. ... rSRP(x)=r1−r2=f_img(x)^T·(C1−C2) ... C1 represents desired attributes (e.g., realistic) and C2 encodes unwanted features."

    SGP-HPS is exactly the SRPO objective for the realism variant: both are HPSv2.1 under a positive control ('Realistic photo') minus HPSv2.1 under a negative control ('CG Render') on the same image. The reported SGP-HPS gain (0.463 to 0.505) is therefore not an external check of realism; it is a measurement of how well the model optimized the very reward SRPO defines. Calling this an evaluation metric is a fitted-input-called-prediction: the input reward difference is being reported as the output evidence.

  2. self definitional [Sec. 3.1, Eqs. (3)-(4) and following text]
    "xt = αt x0 + σt ϵgt (3) x0 = (xt − σt ϵgt)/αt (4) ... A key advantage of this approach is the existence of a closed-form solution, derived from Eq. 4, which can directly recover the clean image from this noisy state."

    Equation (4) is the algebraic inverse of Equation (3) using the same ground-truth noise ϵgt that was just injected. The 'single-step image recovery' is therefore exact by construction; the model is not predicting the denoised image. The claimed 'high accuracy even at early high-noise timesteps' is tautological because the recovery operation is defined to invert the injection. This is a legitimate training mechanism, but it is not a derived empirical result about the model's generation ability.

full rationale

The paper's headline 'over 3x' improvement in human-evaluated realism and aesthetics is supported by a separate human study whose rubric is not identical to the training reward, so the central claim has independent content. However, two places reduce by construction. First, the automatic metric SGP-HPS is the same scalar as the SRPO reward with the realism control words; reporting its improvement is reporting the training objective, not an independent validation. Second, Direct-Align's exact single-step recovery is an algebraic identity: Eq. (4) inverts Eq. (3) with the same injected noise, so the early-timestep recovery accuracy is definitional. These are genuine tautological/circular elements, but they affect a secondary automatic metric and a mechanism description rather than the human-evaluation claim itself. The paper's stated limitations—control tokens may be out-of-domain and text-embedding effects may not align with intent—are honest caveats, not circularity. Self-citations to Hunyuan-DiT and Pref-GRPO are not load-bearing. The human study's absence of inter-annotator agreement and confidence intervals is a validity concern, not circularity. Overall: partial circularity, not a fully forced result.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper's central claims depend on several hand-chosen parameters (lambda, Delta-sigma, control-word pairs) and on assumptions about the reward model's bilinear structure and the validity of the human evaluation. The Direct-Align recovery itself is an algebraic identity and adds no new axiomatic content.

free parameters (5)
  • timestep discount lambda(t) = not reported
    Decaying discount used in Eq. 6 to downweight late timesteps; its shape is hand-chosen and not specified, yet the ablation "w/o discount" shows it changes reward-hacking behavior.
  • Delta-sigma (model-prediction weight) = 0.075 and 0.025 shown in Fig. 3; main value not reported
    Controls how much of the recovered image uses the model's predicted noise vs. the ground-truth noise in Eq. 5; the paper shows final quality depends on this weight.
  • control word pair (C1/C2) = "Realistic photo" vs "CG Render" for main realism results; exact templates not listed for all experiments
    The positive/negative text prompts define the relative reward in Eqs. 9-11; results are sensitive to the reward model's recognition of these words (Sec. 4.5, Suppl. S3).
  • CFG scale k for alternative reward = not reported
    In Eq. 11, the classifier-free-guidance-style blend coefficient for the positive and negative embeddings is not specified; used as an alternative formulation.
  • number and schedule of noise injections n = not reported
    Eq. 6 aggregates rewards over a sequence of injected-noise states; the length and timestep spacing are not specified.
axioms (4)
  • standard math The forward diffusion/flow process is linear: x_t = alpha_t x_0 + sigma_t epsilon
    Invoked in Eqs. 3-4 as the basis of Direct-Align; standard for DDPM and flow matching (cited refs).
  • domain assumption Reward model score is proportional to the inner product of image and text embeddings
    Eq. 7 assumes CLIP-style bilinear reward structure; true of HPSv2.1, PickScore, and CLIP, but not of all reward models.
  • domain assumption Text-prompt conditioning can shift reward direction without touching image-branch gradients
    Sec. 3.2 states bias originates from the image branch and that control words modify C, so only the text embedding shifts; relies on the reward model's text encoder having semantic control.
  • domain assumption The 500-prompt, 10-annotator human protocol is a valid and sufficiently precise measure of realism and aesthetics
    The central "over 3x" claim rests on this evaluation; no inter-annotator agreement or confidence intervals are reported.

pith-pipeline@v1.3.0-alltime-deepseek · 13887 in / 17993 out tokens · 182482 ms · 2026-08-04T22:51:40.967741+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference." pith.science (2026). https://pith.science/paper/IDXTOZHB

@misc{pith2026250906942,
  author       = {Pith},
  title        = {Pith review of: Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IDXTOZHB}},
  note         = {Machine review of arXiv:2509.06942}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent studies have demonstrated the effectiveness of directly aligning diffusion models with human preferences using differentiable reward. However, they exhibit two primary challenges: (1) they rely on multistep denoising with gradient computation for reward scoring, which is computationally expensive, thus restricting optimization to only a few diffusion steps; (2) they often need continuous offline adaptation of reward models in order to achieve desired aesthetic quality, such as photorealism or precise lighting effects. To address the limitation of multistep denoising, we propose Direct-Align, a method that predefines a noise prior to effectively recover original images from any time steps via interpolation, leveraging the equation that diffusion states are interpolations between noise and target images, which effectively avoids over-optimization in late timesteps. Furthermore, we introduce Semantic Relative Preference Optimization (SRPO), in which rewards are formulated as text-conditioned signals. This approach enables online adjustment of rewards in response to positive and negative prompt augmentation, thereby reducing the reliance on offline reward fine-tuning. By fine-tuning the FLUX model with optimized denoising and online reward adjustment, we improve its human-evaluated realism and aesthetic quality by over 3x.

Figures

Figures reproduced from arXiv: 2509.06942 by Chunyu Wang, Donghao Li, Qinglin Lu, Shiyi Zhang, Xiangwei Shen, Yansong Tang, Yingfang Zhang, Zhantao Yang, Zhimin Li.

Figure 1
Figure 1. Figure 1: Images generated by FLUX.1-dev finetuned through our Semantic Relative Preference Optimization (SRPO) Our method substantially improves upon the baseline model, achieving superior photorealism and enhanced fine-grained detail while maintaining remarkable training efficiency-converging in just 10 minutes using 32 NVIDIA H20 GPUs. Abstract Recent studies have demonstrated the effectiveness of di￾rectly align… view at source ↗
Figure 2
Figure 2. Figure 2: Method Overview. The SRPO contains two key elements: Direct-Align, and a single reward model that derives both rewards and penalties from positive and negative prompts. The pipeline of Direct-Align consists of four stages: (0) generate/load an image for training; (1) inject noise into image; (2) perform one-step denoise/inversion; (3) recover image. 2. Related Work Optimization on Diffusion Timesteps. Rece… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison on one-step prediction at early timestep The values 0.075 and 0.025 denote the weight of the model predic￾tion term used for method, respectively. The earliest 5% represent state with 95% noise from an unshifted timestep. By constructing a Gaussian prior, our one-step sampling method achieves high￾quality results at early timesteps, even when the input image is highly noised. Reward Aggregation … view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of human evaluation results for Vanilla FLUX, ReFL, DRaFT LV, DanceGRPO, Direct-Align, and SRPO on the criteria of Realism, Aesthetics, and Overall Preference. SRPO demonstrates significant improvements in Aesthetics and achieves a substantial reduction in AIGC artifacts. crossed experimental design. The assessment focuses on four critical dimensions of image quality: (1) Text-image alignment (s… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative Comparison on FLUX, DanceGRPO and SRPO with same seed. Our approach demonstrates superior perfor￾mance in realism and detail complexity. FLUX SRPO(CLIP) SRPO(Pick) SRPO(HPS） SRPO(AE） [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cross-reward results of SRPO. cluding CLIP, though improvements with CLIP remain lim￾ited due to its lack of human preference alignment. Notably, PickScore demonstrates faster and more stable convergence than HPS, while both yield comparable visual quality. Cru￾cially, no reward hacking is observed in our method, high￾lighting the effectiveness of Direct-Align’s design ( [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of Optimization Effects of Differ￾ent timestpe Intervals & Comparison of Reward-System and SRPO on Direct-Align. (1) Hacking Rate: Annotators compare three outputs and select the one that is least detailed or most over￾processed, labeling it as hacking (2) The prompt is A young girl riding a gray wolf in a dark forest. Reward-System can only ad￾justs scale of rewards, resulting in trade-offs bet… view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of SRPO-controlled results for different style words [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Overview of experimental results demonstrating the key properties of our SRPO method on the HPDv2 dataset: A: Com￾parison between FLUX.1.Krea and FLUX.1.dev following the application of our SRPO method. B: Comparison between our main model and vanilla FLUX.1.dev using realism-related recaptioning. C: Illustration of enhanced style control achieved through the incorporation of style-word conditioning. D: Ab… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. LeapAlign: Post-Training Flow Matching Models at Any Generation Step by Building Two-Step Trajectories

    cs.CV 2026-04 unverdicted novelty 7.0

    LeapAlign fine-tunes flow matching models by constructing two consecutive leaps that skip multiple ODE steps with randomized timesteps and consistency weighting, enabling stable updates at any generation step.

  2. WearWow: Native 2K Multi-Garment Virtual Try-On via Adaptive Token Packing and Preference Alignment

    cs.CV 2026-07 conditional novelty 6.0

    WearWow generates native 2K multi-garment virtual try-on images without masks, using token packing plus dual preference rewards to preserve fabric texture.

  3. TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment

    cs.CV 2026-01 conditional novelty 6.0

    TAGRPO adds a trajectory-alignment loss and a memory bank to GRPO, improving reward-model scores over DanceGRPO for image-to-video generation on two modern backbones.

  4. Distribution Matching Distillation Meets Reinforcement Learning

    cs.CV 2025-11 conditional novelty 6.0

    Combining DMD distillation with RL during training produces few-step text-to-image models that outperform their multi-step teacher on several benchmarks.

  5. HunyuanImage 3.0 Technical Report

    cs.CV 2025-09 accept novelty 6.0

    HunyuanImage 3.0 delivers an 80B-parameter MoE model unifying multimodal understanding and generation that matches prior state-of-the-art results while being fully open-sourced.

  6. HunyuanImage 3.0 Technical Report

    cs.CV 2025-09 conditional novelty 6.0

    HunyuanImage 3.0 is an open 80B-parameter multimodal autoregressive image generator that reportedly matches leading closed models on in-house benchmarks.

  7. When Preference Labels Fall Short: Aligning Diffusion Models from Real Data

    cs.CV 2026-05 unverdicted novelty 5.0

    Real-data supervision via reference-based contrasts aligns diffusion models comparably to standard preference methods without annotated pairs.

  8. When Preference Labels Fall Short: Aligning Diffusion Models from Real Data

    cs.CV 2026-05 unverdicted novelty 5.0

    Real images contrasted with generated samples can supply effective preference signals for aligning diffusion models at performance levels comparable to standard preference-pair methods.

  9. FAIL: Flow Matching Adversarial Imitation Learning for Image Generation

    cs.CV 2026-02 conditional novelty 5.0

    Post-training of flow matching can be framed as adversarial imitation learning, and the proposed FAIL methods improve FLUX's generation quality using 13K expert images without preference pairs.

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages · cited by 7 Pith papers

  1. [1]

    com / discus0434/aesthetic-predictor-v2-5, 2025

    Aesthetic predictor v2.5.https : / / github . com / discus0434/aesthetic-predictor-v2-5, 2025. Accessed: 2025-06-10. 2, 5, 7

    work page 2025

  2. [2]

    Stochastic interpolants: A unifying framework for flows and diffusions.arXiv preprint arXiv:2303.08797,

    Michael S Albergo, Nicholas M Boffi, and Eric Vanden- Eijnden. Stochastic interpolants: A unifying framework for flows and diffusions.arXiv preprint arXiv:2303.08797,

    Pith/arXiv arXiv

  3. [3]

    Enhancing reward models for high-quality image generation: Beyond text-image align- ment.arXiv preprint arXiv:2507.19002, 2025

    Ying Ba, Tianyu Zhang, Yalong Bai, Wenyi Mo, Tao Liang, Bing Su, and Ji-Rong Wen. Enhancing reward models for high-quality image generation: Beyond text-image align- ment.arXiv preprint arXiv:2507.19002, 2025. 2, 3

    Pith/arXiv arXiv 2025

  4. [4]

    Training diffusion models with reinforce- ment learning.arXiv preprint arXiv:2305.13301, 2023

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforce- ment learning.arXiv preprint arXiv:2305.13301, 2023. 2

    Pith/arXiv arXiv 2023

  5. [5]

    Directly fine-tuning diffusion models on differentiable re- wards.arXiv preprint arXiv:2309.17400, 2023

    Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable re- wards.arXiv preprint arXiv:2309.17400, 2023. 2, 3, 4

    Pith/arXiv arXiv 2023

  6. [6]

    Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic op- timal control.arXiv preprint arXiv:2409.08861, 2024

    Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic op- timal control.arXiv preprint arXiv:2409.08861, 2024. 2, 3

    Pith/arXiv arXiv 2024

  7. [7]

    Optimizing ddpm sampling with shortcut fine-tuning.arXiv preprint arXiv:2301.13362,

    Ying Fan and Kangwook Lee. Optimizing ddpm sampling with shortcut fine-tuning.arXiv preprint arXiv:2301.13362,

    Pith/arXiv arXiv

  8. [8]

    Dpok: Reinforcement learning for fine-tuning text-to-image diffu- sion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffu- sion models.Advances in Neural Information Processing Systems, 36:79858–79885, 2023. 2

    2023

  9. [9]

    Geneval: An object-focused framework for evaluating text- to-image alignment.Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text- to-image alignment.Advances in Neural Information Pro- cessing Systems, 36:52132–52152, 2023. 2, 5

    work page 2023

  10. [10]

    Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022. 5

    Pith/arXiv arXiv 2022

  11. [11]

    Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 2, 3, 7

    2020

  12. [12]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation

    Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Ma- tiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 36652–36663, 2023. 2, 3, 5

    work page 2023

  13. [13]

    Flux.https://github.com/ black-forest-labs/flux, 2024

    Black Forest Labs. Flux.https://github.com/ black-forest-labs/flux, 2024. 2, 5

    work page 2024

  14. [14]

    Aligning text- to-image models using human feedback.arXiv preprint arXiv:2302.12192, 2023

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text- to-image models using human feedback.arXiv preprint arXiv:2302.12192, 2023. 2

    Pith/arXiv arXiv 2023

  15. [15]

    Flux.1 krea [dev].https://github.com/krea-ai/flux- krea, 2025

    Sangwu Lee, Titus Ebbecke, Erwann Millon, Will Beddow, Le Zhuo, Iker Garc´ıa-Ferrero, Liam Esparraguera, Mihai Pe- trescu, Gian Saß, Gabriel Menezes, and Victor Perez. Flux.1 krea [dev].https://github.com/krea-ai/flux- krea, 2025. 2, 5

    work page 2025

  16. [16]

    Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding.arXiv preprint arXiv:2405.08748, 2024

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chi- nese understanding.arXiv preprint arXiv:2405.08748, 2024. 3

    Pith/arXiv arXiv 2024

  17. [17]

    Aes- thetic post-training diffusion models from generic prefer- ences with step-by-step preference optimization

    Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aes- thetic post-training diffusion models from generic prefer- ences with step-by-step preference optimization. InProceed- ings of the Computer Vision and Pattern Recognition Confer- ence, pages 13199–13208, 2025. 3

    work page 2025

  18. [18]

    Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximil- ian Nickel, and Matt Le. Flow matching for generative mod- eling.arXiv preprint arXiv:2210.02747, 2022. 3

    Pith/arXiv arXiv 2022

  19. [19]

    Flow-grpo: Training flow matching models via on- line rl.arXiv preprint arXiv:2505.05470, 2025

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via on- line rl.arXiv preprint arXiv:2505.05470, 2025. 2

    Pith/arXiv arXiv 2025

  20. [20]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 3

    Pith/arXiv arXiv 2022

  21. [21]

    Sit: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers

    Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Explor- ing flow and diffusion-based generative models with scalable interpolant transformers. InEuropean Conference on Com- puter Vision, pages 23–40. Springer, 2024. 3

    work page 2024

  22. [22]

    Hpsv3: Towards wide-spectrum human preference score

    Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. arXiv preprint arXiv:2508.03789, 2025. 3

    Pith/arXiv arXiv 2025

  23. [23]

    The ef- fects of reward misspecification: Mapping and mitigating misaligned models.arXiv preprint arXiv:2201.03544, 2022

    Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The ef- fects of reward misspecification: Mapping and mitigating misaligned models.arXiv preprint arXiv:2201.03544, 2022. 2

    Pith/arXiv arXiv 2022

  24. [24]

    Aligning text-to-image diffusion models with reward backpropagation

    Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Ka- terina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. 2023. 2, 3, 4

    work page 2023

  25. [25]

    Video diffusion align- ment via reward gradients.arXiv preprint arXiv:2407.08737,

    Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Kate- rina Fragkiadaki, and Deepak Pathak. Video diffusion align- ment via reward gradients.arXiv preprint arXiv:2407.08737,

    Pith/arXiv arXiv

  26. [26]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021. 4

    2021

  27. [27]

    Laion-aesthetics.https : / / laion.ai/blog/laion- aesthetics/, 2022

    Christoph Schuhmann. Laion-aesthetics.https : / / laion.ai/blog/laion- aesthetics/, 2022. Ac- cessed: 2023-11-10. 3, 5

    work page 2022

  28. [28]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020. 2, 3

    Pith/arXiv arXiv 2010

  29. [29]

    Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions.arXiv preprint arXiv:2011.13456, 2020. 3

    Pith/arXiv arXiv 2011

  30. [30]

    Pref-grpo: Pairwise preference reward-based grpo for sta- ble text-to-image reinforcement learning.arXiv preprint arXiv:2508.20751, 2025

    Yibin Wang, Zhimin Li, Yuhang Zang, Yujie Zhou, Jiazi Bu, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Pref-grpo: Pairwise preference reward-based grpo for sta- ble text-to-image reinforcement learning.arXiv preprint arXiv:2508.20751, 2025. 2

    Pith/arXiv arXiv 2025

  31. [31]

    Diffu- siondb: A large-scale prompt gallery dataset for text-to- image generative models.arXiv preprint arXiv:2210.14896,

    Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffu- siondb: A large-scale prompt gallery dataset for text-to- image generative models.arXiv preprint arXiv:2210.14896,

    Pith/arXiv arXiv

  32. [32]

    Human preference score: Better aligning text- to-image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hong- sheng Li. Human preference score: Better aligning text- to-image models with human preference. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 2096–2105, 2023. 2, 3, 5

    work page 2096

  33. [33]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation.Advances in Neural Information Pro- cessing Systems, 36:15903–15935, 2023

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation.Advances in Neural Information Pro- cessing Systems, 36:15903–15935, 2023. 2, 3, 4, 5, 8

    work page 2023

  34. [34]

    Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818, 2025

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818, 2025. 2, 5

    Pith/arXiv arXiv 2025

  35. [35]

    Teaching large language models to regress accurate image quality scores using score distribution.arXiv preprint arXiv:2501.11561, 2025

    Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching large language models to regress accurate image quality scores using score distribution.arXiv preprint arXiv:2501.11561, 2025. 5

    arXiv 2025

  36. [36]

    Learning multi- dimensional human preference for text-to-image generation

    Sixian Zhang, Bohan Wang, Junqiang Wu, Yan Li, Tingt- ing Gao, Di Zhang, and Zhongyuan Wang. Learning multi- dimensional human preference for text-to-image generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8018–8027, 2024. 3

    work page 2024

  37. [37]

    Diffusion model as a noise-aware latent reward model for step-level preference optimization.arXiv preprint arXiv:2502.01051, 2025

    Tao Zhang, Cheng Da, Kun Ding, Huan Yang, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chun- hong Pan. Diffusion model as a noise-aware latent reward model for step-level preference optimization.arXiv preprint arXiv:2502.01051, 2025. 3 Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference Supplementary Material ...

    arXiv 2025