Pith. sign in

REVIEW 5 major objections 5 minor 5 cited by

By aligning reasoning representations with discriminative features in latent space, chain-of-thought reasoning can be delivered through a fast scalar reward signal.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:33 UTC pith:QAC6KKUD

load-bearing objection A genuinely promising training recipe for discriminative visual reward models, with SOTA claims that hinge on undisclosed train/eval split details and a mechanism story backed by indirect evidence. the 5 major comments →

arxiv 2602.07533 v3 pith:QAC6KKUD submitted 2026-02-07 cs.AI

Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models

Yankai Yang , Yancheng Long , Hongyang Wei , Wei Chen , Tianke Zhang , Kaiyu Jiang , Haonan Fan , Changyi Liu
show 7 more authors
Jiankang Chen Kaiyu Tang Bin Wen Fan Yang Tingting Gao Han Li Shuo Yang
This is my paper
classification cs.AI
keywords reward modelingreinforcement learning from human feedbackimage editingchain-of-thoughtlatent reasoningvision-language modelpreference learningjoint training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to resolve a practical trade-off in reward modeling for image editing: models trained to rank images against human preference are fast but shallow, while models that reason in text before scoring are deep but slow and expensive. The proposed method, Joint Reward Modeling (JRM), trains one shared vision-language backbone on both a preference ranking loss and a language modeling loss, using structured written explanations as extra supervision. The central claim is that the reasoning competence can live inside the shared hidden representation, so at inference time the language generator is switched off and a lightweight score head reads the same quality of judgment directly. On two public benchmarks the authors report the best accuracy among compared methods, and in downstream online reinforcement learning this reward signal yields the largest gains, all at zero test-time reasoning cost. If right, this means fast reward models do not have to sacrifice deep semantic evaluation.

Core claim

JRM's central claim is that generative-style reasoning can be internalized into the latent space of a discriminative reward model. A shared representation h = E(x, c) is jointly constrained by an uncertainty-aware ranking loss over human preferences and a cross-entropy language loss over structured evaluation text. During inference the language head is removed, and the scalar score head reads directly from h; the paper calls this implicit process Latent Chain-of-Thought. The supporting evidence is a 85.1% overall score on EditReward-Bench and 69.3% on MMRB2, a near-doubling of effective feature-space rank versus the ranking-only baseline (91.77 vs 46.86), and downstream reinforcement-learnin

What carries the argument

The load-bearing mechanism is the shared representation h produced by the vision-language backbone for an image pair and instruction. Two heads read from h: a RankNet-style discriminative score head trained with an uncertainty-aware Gaussian ranking loss, and a conditional language head trained to generate structured, region-referencing evaluation text. The language loss pushes h to encode global semantic structure, while the ranking loss keeps h ordered by human preference. At test time only the score head runs, so the claimed reasoning gain is obtained without generating text.

Load-bearing premise

The argument depends on the generated explanations used as language supervision carrying semantic information beyond the pre-assigned score they were prompted with; if the text mostly restates the score, the language loss adds no new reasoning signal and the claimed internalization reduces to extra regularization.

What would settle it

Train JRM exactly as described but replace each explanation with a length-matched, score-matched explanation from a different sample, keeping the ranking loss and data volume fixed. If benchmark accuracy stays at JRM's reported level, the language objective is contributing regularization or compute rather than reasoning; if accuracy drops, the semantic content of the explanations is doing the work. A complementary check is to evaluate JRM on instructions and image pairs with no overlap with its training distribution.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reward models for image editing no longer need to generate reasoning at evaluation time: JRM reports 85.1% on EditReward-Bench and 69.3% on MMRB2 while using the same discriminative inference path as a plain ranking scorer.
  • Using JRM as the reward signal in online reinforcement learning improves an image-editing model by +1.00 on GEdit-Bench and +0.50 on ImageEdit-Bench, larger than the gains reported for the compared closed-source judges.
  • Joint language supervision stabilizes and accelerates ranking-loss convergence and roughly doubles the effective rank of the shared representation, which the paper interprets as preventing collapse to shallow visual cues.
  • In the tested range, larger language-supervision weight α consistently improves benchmark accuracy, suggesting the language objective contributes semantic structure rather than acting only as a regularizer.
  • The language head can be re-enabled as a diagnostic to produce textual evaluations whose stated problems match the scalar scores, enabling JRM-guided self-correction of editing models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism transfers, the same joint ranking-plus-language objective could produce fast reasoning-like discriminators in other expensive evaluation domains, such as video generation or multi-turn dialogue, where full chain-of-thought generation is too costly at test time.
  • The decisive untested control is to train JRM with the same generated explanations but with their content shuffled relative to the scores; if accuracy survives the shuffle, the gain comes from extra data or compute rather than from the semantics of the reasoning text.
  • The sharp rise in effective rank suggests a practical extension the paper does not run: after joint training, prune or distill the language head entirely and check how much of the benchmark advantage remains in the score head alone.
  • The region-tagged language supervision hints that the shared representation could be used not only to score an edit but to localize what went wrong, potentially supporting targeted repair rather than reward assignment alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Joint Reward Modeling (JRM), a method that trains a shared vision-language backbone with both a preference-ranking loss and a language-modeling loss on image-editing data, then uses only the discriminative scoring head at inference. The central claim is that joint training internalizes chain-of-thought-style reasoning into the shared representation, so that a discriminative reward model can match or exceed generative reward models in semantic evaluation without paying generation cost at inference. The authors report state-of-the-art accuracy on EditReward-Bench (85.1% vs GPT-5's 75.5%) and MMRB2 (69.3% vs 61.9%), representation-space evidence (effective rank 91.77 vs 46.86), and downstream online RL gains with JRM as the reward signal on GEdit-Bench (+1.00) and ImageEdit-Bench (+0.50).

Significance. If the empirical claims hold, JRM would be a practically valuable result: it would show that language-supervision can be folded into a discriminative reward model, combining reasoning-like semantic understanding with efficient inference suitable for online RL. The paper has clear strengths: the method is simple and reproducible in principle, the authors provide a code link, they evaluate on multiple benchmarks, ablate the language-supervision weight, analyze internal representations, and test the model in an actual Flow-GRPO loop. However, the current evidence is not yet sufficient to support the central SOTA and mechanism claims, because of a likely train/evaluation overlap and the lack of a control isolating the information added by the generated language targets.

major comments (5)
  1. [§3.1 + Table 1] The training data is built 'upon existing image editing reward datasets (Wu et al., 2025d)', which is the same source that introduces EditReward-Bench, the benchmark in Table 1. The paper never discloses whether EditReward-Bench evaluation pairs were excluded from training, whether duplicates or near-duplicates were removed, or what the exact train/eval split is. If any evaluation pairs or images from the same editing tasks appear in the training corpus, the reported 85.1% Overall and 85.4% Prompt-Following numbers could be inflated by memorization rather than by internalized reasoning. The same corpus is also used to train the reward signal for the downstream RL experiments, so contamination could affect the GEdit +1.00 / ImageEdit +0.50 gains. This is a load-bearing disclosure that must be provided.
  2. [Appendix A.1 + §3.2] The language-supervision targets are generated by prompting a VLM with the pre-assigned score as part of the input ('generate a structured evaluation entry for an image editing pair based on a pre-assigned score'). The LM loss may therefore simply learn to verbalize the scalar score, adding no semantic information beyond what the ranking loss already provides. The central mechanism claim — that reasoning is 'internalized' into the representation — requires a control that isolates the information content of the language targets. For example, training with score-scrambled or paraphrased language targets, or with language targets produced without conditioning on the score, would show whether the effect comes from the language content or from additional regularization/compute. Without this control, the 'internalized reasoning' claim is not established.
  3. [§4.2, Figure 4] The note under Figure 4 states that except for α=0.7, the optimal checkpoint differs between the two benchmarks, and results shown are the best performance achieved on each benchmark. This makes the α-ablation an upper envelope, not a fair comparison of a single training run, and it means α=0.7 is selected post hoc for each benchmark. The monotonic improvement trend claimed in the text is therefore not supported as stated. Reporting a single held-out checkpoint, or multiple seeds with error bars, would be needed to support the claim that language supervision has a monotonic positive effect.
  4. [§4.1, Tables 1–2, Table 4] No error bars, confidence intervals, or significance tests are reported for any of the central numbers. Several headline comparisons are small in absolute terms (e.g., MMRB2 Overall 69.3% vs EditReward's 65.7%; EditReward-Bench Overall 85.1% vs PaCo-Reward's 75.1% is larger but still a single run). Since model training is stochastic, the claim of 'significantly outperforms' (Section 1) requires at least multiple seeds or a statement about evaluation/test-set determinism. This is particularly important for the downstream RL results in Table 4, where +0.19 vs +0.23 vs +0.50 are within the range of typical RL-run variance.
  5. [§4.3 + §3.3] The representation analysis shows that joint training changes the singular-value spectrum and effective rank, but it does not establish that the additional dimensions encode reasoning semantics. The comparison is between α=0 and α=0.7; any additional training signal (including the extra language-modeling objective and the associated compute) could produce a more dispersed representation without instantiating a 'Latent Chain-of-Thought'. The paper needs a control (e.g., training with an equally weighted but non-semantic auxiliary loss) to support the mechanistic interpretation in Eq. (8)–(9).
minor comments (5)
  1. [Table 1] The 'Cons.' column has dashes for EditReward and JRM, but the text discusses only PF and Overall. Please clarify what 'Cons.' is, why it is omitted for these two methods, and whether it is a meaningful comparison dimension.
  2. [References] Several references are incomplete: 'Yao et al.' has no year or venue, 'Ankner et al.' has no year, and Zhang et al. appears as 'Zhang et al.;' with no citation details in the Related Work section.
  3. [§3.3, Eq. (9)] The notation 'rank Cov(h) ↑' is informal and not a well-defined equation; please state the intended mathematical claim (e.g., effective rank of the covariance matrix of a batch of representations).
  4. [Figure 5 / Appendix B] The text says 'For training details, see Figure 5 and Appendix B,' but Figure 5 shows loss dynamics, not hyperparameter details. The actual training hyperparameters are in Appendix B; please fix the pointer.
  5. [§3.2, Appendix B] The shared representation h is first defined in Eq. (1) as the output of E(x,c), but the concrete implementation (a learnable special token whose final hidden state is h) appears only in Appendix B. Moving this detail into Section 3.2 would make the architecture clearer.

Circularity Check

0 steps flagged

No significant circularity; the central JRM claim is empirically testable and supported by an external benchmark, with two non-circular data-quality caveats.

full rationale

Walking the derivation chain: Eq. (7) defines the joint objective L_total = (1-alpha)L_rank + alpha L_LM, and inference (Eq. 10) outputs r = f_theta(E(x,c)) from the shared representation only. No predicted quantity is computed from a parameter that was fit to that same quantity; the score head is not a renamed version of the language head or of the language targets. The alpha=0.7 choice is a hyperparameter selected from benchmark curves (Fig. 4) and is normal model selection, not a fitted input renamed as a prediction. The central SOTA claim is not exclusively self-referential: MMRB2 (Hu et al., 2025) is an external benchmark, and JRM beats prior methods there too. The representation-rank analysis (91.77 vs 46.86) is an additional empirical observation, not a tautology. Two caveats are worth flagging but neither is a circular reduction. First, Appendix A.1 generates language supervision by prompting a VLM 'based on a pre-assigned score,' so the L_LM target is partly a verbalization of the same preference label used in L_rank; this is a supervision-redundancy / mechanism-identification concern, not a case where the derivation equals its input by construction. Second, Section 3.1 says JRM is built 'upon existing image editing reward datasets (Wu et al., 2025d),' while Table 1 evaluates EditReward-Bench from the same citation, with no disclosed train/eval split; this is a potential contamination risk that the authors can settle empirically, but absent evidence of overlap it cannot be called circular. Self-citations to R1-Reward / Keye-VL appear as background related work and are not load-bearing for the main result.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 1 invented entities

The method's central claim rests on the quality and informativeness of auto-generated language supervision, on the assumption that representation-space statistics reflect reasoning ability, and on the reliability of the adopted preference datasets and RL algorithm. The only explicit fitted hyperparameter is alpha; however, checkpoint selection per benchmark adds an unquantified selection degree of freedom. The 'Latent CoT' construct is the main invented entity and lacks independent evidence.

free parameters (1)
  • alpha (language supervision weight) = 0.7
    Weight balancing ranking loss and LM loss in L_total (Equation 7); tuned against benchmark performance across values 0, 0.3, 0.5, 0.7 (Section 4.2).
axioms (5)
  • domain assumption Human preference comparisons in EditReward/MMRB2 are reliable ground truth for reward quality.
    Used as supervision labels for L_rank (Section 3.2) and as test labels in Tables 1-2; the paper does not independently audit label noise.
  • domain assumption A VLM prompted with the assigned score produces faithful, informative reasoning text (Appendix A).
    This underpins the entire language-supervision pipeline; the paper does not validate that the generated text carries information beyond the scalar score.
  • domain assumption Effective rank, isotropy, and spectral entropy of hidden representations measure 'reasoning capability'.
    Section 4.3 connects these representation statistics to internalized reasoning without a direct behavioral test of reasoning.
  • standard math The uncertainty-aware ranking formula from HPSv3 (Equations 4-5) models reward noise adequately.
    Borrowed from Ma et al. 2025; used without modification or new derivation.
  • standard math Flow-GRPO (Liu et al., 2025a) provides a valid online RL alignment procedure.
    Used for downstream RL experiments; implementation details in Appendix B.2.
invented entities (1)
  • Latent Chain-of-Thought (Latent CoT) no independent evidence
    purpose: Postulated implicit reasoning process encoded in the shared representation h during joint training; invoked to explain why a discriminative reward head can perform semantic evaluation without generating text (Section 3.3, Equation 8).
    Supported only by indirect internal metrics (effective rank 91.77 vs 46.86, isotropy, spectral entropy) and downstream benchmark gains; there is no direct manipulation or causal test isolating this latent process.

pith-pipeline@v1.3.0-alltime-deepseek · 14738 in / 14031 out tokens · 135273 ms · 2026-08-03T03:33:24.319740+00:00 · methodology

0 comments
read the original abstract

Reward models are critical for reinforcement learning from human feedback, as they determine the alignment quality and reliability of generative models. For complex tasks such as image editing, reward models are required to capture global semantic consistency and implicit logical constraints beyond local similarity. Existing reward modeling approaches have clear limitations. Discriminative reward models align well with human preferences but struggle with complex semantics due to limited reasoning supervision. Generative reward models offer stronger semantic understanding and reasoning, but they are costly at inference time and difficult to align directly with human preferences. To this end, we propose Joint Reward Modeling (JRM), which jointly optimizes preference learning and language modeling on a shared vision-language backbone. This approach internalizes the semantic and reasoning capabilities of generative models into efficient discriminative representations, enabling fast and accurate evaluation. JRM achieves state-of-the-art results on MMRB2 and EditReward-Bench, and significantly improves stability and performance in downstream online reinforcement learning. These results show that joint training effectively bridges efficiency and semantic understanding in reward modeling.

Figures

Figures reproduced from arXiv: 2602.07533 by Bin Wen, Changyi Liu, Fan Yang, Han Li, Haonan Fan, Hongyang Wei, Jiankang Chen, Kaiyu Jiang, Kaiyu Tang, Shuo Yang, Tianke Zhang, Tingting Gao, Wei Chen, Yancheng Long, Yankai Yang.

Figure 1
Figure 1. Figure 1: Performance comparison. Left: Comparison of Genera￾tive, Discriminative, and Joint Reward Modeling (JRM) paradigms. Right: JRM achieves state-of-the-art accuracy on benchmarks and significantly boosts downstream RL performance. Note: in the performance comparison, Generative RM refers to EditScore (Luo et al., 2025), and Discriminative RM refers to EditReward (Wu et al., 2025d). 2025a; Xue et al., 2025). T… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the JRM Framework. The workflow consists of three stages: (1) Joint Training, where the model internalizes reasoning capabilities; (2) Efficient Inference, which retains only the discriminative pathway; and (3) the Online RL Loop, where JRM provides scalable feedback for downstream alignment. soning supervision, discriminative reward models often rely on local visual patterns or shallow sta… view at source ↗
Figure 3
Figure 3. Figure 3: Attention visualization. Compared to the baseline, JRM accurately focuses on salient regions specified by editing instructions, where the baseline refers to a discriminative reward model without semantic supervision. Bench (Wu et al., 2025d), JRM achieves an accuracy of 85.1%, outperforming GPT-5 (75.5%) by 9.6%. On the MMRB2 (Hu et al., 2025) benchmark, JRM reaches a com￾posite score of 69.3%, exceeding G… view at source ↗
Figure 5
Figure 5. Figure 5: illustrates the training dynamics of both loss com￾ponents. Notably, when α > 0 (with language supervision), the ranking loss converges faster and more stably than the baseline (α = 0), suggesting that language supervision pro￾vides beneficial inductive bias for reward learning. Mean￾while, the cross-entropy loss converges smoothly, indicating that the two objectives are complementary rather than con￾flict… view at source ↗
Figure 4
Figure 4. Figure 4: Impact of different language supervision weights α on model performance. As α increases, model performance steadily improves on both benchmarks. Note: except for α = 0.7 (the chosen setting for JRM), the optimal checkpoints differ between the two benchmarks; results shown represent the best performance achieved on each benchmark respectively. To analyze the impact of language supervision signals on reward … view at source ↗
Figure 6
Figure 6. Figure 6: Representation space analysis. Left: Singular value spectrum comparison: JRM (blue) exhibits a significantly flatter decay compared to the baseline (red), indicating higher-dimensional information encoding. Middle: Representation quality metrics: JRM achieves substantially higher effective rank (91.77 vs. 46.86), spectral entropy, and isotropy score, confirming that joint training prevents representation c… view at source ↗
Figure 7
Figure 7. Figure 7 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative comparison of RL-finetuned models. Om￾niGen2 aligned with JRM demonstrates superior semantic under￾standing. JRM exhibits a steady increase during RL training, and the evaluation scores on GEdit-Bench closely align with the reward curves. This strong correlation validates the accu￾racy and reliability of JRM as a reward model. For more training details, see Appendix B [PITH_FULL_IMAGE:figures/… view at source ↗
Figure 10
Figure 10. Figure 10: Gradient norm stability during JRM training. Higher α values lead to larger initial gradients but converge rapidly within the first epoch [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Late-stage loss convergence (epochs 2–5.5). All configurations show stable convergence, with lower α achieving lower final loss values. B.2. Reinforcement Learning Fine-Tuning Training Hyperparameters. For online RL fine-tuning of OmniGen2-Edit, we use the Flow-GRPO algorithm. • Algorithm: Flow-GRPO (Group Relative Policy Optimization) • Discrete Timesteps T: 20 • Diffusion Coefficient σ: 0.9 • Global Bat… view at source ↗
Figure 12
Figure 12. Figure 12: Loss components during Flow-GRPO training. The figure shows the dynamics of individual loss terms during online RL fine-tuning with JRM as the reward signal. C. JRM Inference Prompts At inference time, JRM can optionally generate language evaluations through the language head. Below are the complete prompt templates used for each evaluation dimension. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Qualitative comparison of RL-fine-tuned OmniGen2 models (Part 1/4). From left to right: Source Image, base OmniGen2, OmniGen2 fine-tuned with baseline reward model (w/ Baseline-RL), and OmniGen2 fine-tuned with JRM (w/ JRM). 19 [PITH_FULL_IMAGE:figures/full_fig_p019_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Qualitative comparison of RL-fine-tuned OmniGen2 models (Part 2/4). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Qualitative comparison of RL-fine-tuned OmniGen2 models (Part 3/4). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Qualitative comparison of RL-fine-tuned OmniGen2 models (Part 4/4). The JRM-guided model demonstrates superior instruction following and visual quality across diverse editing tasks. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Will It Go Viral? Grounding Micro-Video Popularity Prediction on the Open Web

    cs.MM 2026-05 unverdicted novelty 7.0

    WEBSHORTS dataset and SHORTS-CAST framework ground micro-video popularity prediction in structured open-web context collected at upload time and enable selective online adaptation using delayed labels.

  2. SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing

    cs.CV 2026-06 unverdicted novelty 6.0

    SpatialFlow-GRPO improves image editing quality by converting region-aware rewards into semantic-region-level optimization signals aligned with latent positions during policy updates.

  3. SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing

    cs.CV 2026-06 unverdicted novelty 6.0

    SpatialFlow-GRPO adds region-level reward feedback and spatial alignment to Flow-GRPO-style RL for image editing, reporting gains on GEdit-Bench, ImgEdit-Bench, and a new MultiEditBench.

  4. Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

    cs.CV 2026-06 unverdicted novelty 6.0

    Z-Reward trains a 27B reasoning teacher VLM on score distributions via GDSO and distills it via RISD into a 9B student, reaching 89.6% and 88.6% human preference accuracy with 41.3% optimization gain over SFT baseline.

  5. Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributions

    cs.CV 2026-06 conditional novelty 6.0

    A teacher-student reward model learns reasoning-conditioned score distributions for text-to-image images, yielding ~89% preference accuracy and a 41% net human-preference gain when used for generator optimization.

Reference graph

Works this paper leans on

20 extracted references · 16 linked inside Pith · cited by 3 Pith papers

  1. [4]

    Emerging proper- ties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    Deng, C., Zhu, D., Li, K., Gou, C., Li, F., Wang, Z., Zhong, S., Yu, W., Nie, X., Song, Z., et al. Emerging proper- ties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683,

    Pith/arXiv arXiv

  2. [5]

    Onereward: Unified mask-guided image generation via multi-task human preference learning.arXiv preprint arXiv:2508.21066,

    Gong, Y ., Wang, X., Wu, J., Wang, S., Wang, Y ., and Wu, X. Onereward: Unified mask-guided image generation via multi-task human preference learning.arXiv preprint arXiv:2508.21066,

    Pith/arXiv arXiv

  3. [6]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  4. [7]

    Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324,

    He, X., Fu, S., Zhao, Y ., Li, W., Yang, J., Yin, D., Rao, F., and Zhang, B. Tempflow-grpo: When timing matters for grpo in flow models.arXiv preprint arXiv:2508.04324,

    Pith/arXiv arXiv

  5. [8]

    Multimodal reward- bench 2: Evaluating omni reward models for interleaved text and image.arXiv preprint arXiv:2512.16899,

    Hu, Y ., Askari-Hemmat, R., Hall, M., Dinan, E., Zettle- moyer, L., and Ghazvininejad, M. Multimodal reward- bench 2: Evaluating omni reward models for interleaved text and image.arXiv preprint arXiv:2512.16899,

    arXiv

  6. [10]

    Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025a

    Liu, J., Liu, G., Liang, J., Li, Y ., Liu, J., Wang, X., Wan, P., Zhang, D., and Ouyang, W. Flow-grpo: Training flow matching models via online rl.arXiv preprint arXiv:2505.05470, 2025a. Liu, S., Han, Y ., Xing, P., Yin, F., Wang, R., Cheng, W., Liao, J., Wang, Y ., Fu, H., Han, C., et al. Step1x-edit: A practical framework for general image editing.arXiv...

    Pith/arXiv arXiv

  7. [11]

    Editscore: Unlocking online rl for image editing via high-fidelity reward modeling

    Luo, X., Wang, J., Wu, C., Xiao, S., Jiang, X., Lian, D., Zhang, J., Liu, D., et al. Editscore: Unlocking online rl for image editing via high-fidelity reward modeling. arXiv preprint arXiv:2509.23909,

    arXiv

  8. [13]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  9. [14]

    See- dream 4.0: Toward next-generation multimodal image generation.arXiv preprint arXiv:2509.20427,

    Seedream, T., Chen, Y ., Gao, Y ., Gong, L., Guo, M., Guo, Q., Guo, Z., Hou, X., Huang, W., Huang, Y ., et al. See- dream 4.0: Toward next-generation multimodal image generation.arXiv preprint arXiv:2509.20427,

    Pith/arXiv arXiv

  10. [15]

    Unified multimodal chain-of-thought reward model through reinforcement fine-tuning.arXiv preprint arXiv:2505.03318, 2025a

    Wang, Y ., Li, Z., Zang, Y ., Wang, C., Lu, Q., Jin, C., and Wang, J. Unified multimodal chain-of-thought reward model through reinforcement fine-tuning.arXiv preprint arXiv:2505.03318, 2025a. Wang, Y ., Zang, Y ., Li, H., Jin, C., and Wang, J. Unified re- ward model for multimodal understanding and generation. CoRR, 2025b. Wei, H., Xu, B., Liu, H., Wu, C...

    arXiv

  11. [16]

    Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025a

    Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.-m., Bai, S., Xu, X., Chen, Y ., et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324, 2025a. Wu, C., Zheng, P., Yan, R., Xiao, S., Luo, X., Wang, Y ., Li, W., Jiang, X., Liu, Y ., Zhou, J., et al. Omnigen2: Exploration to advanced multimodal generation.arXiv preprint arXiv:2506.188...

    Pith/arXiv arXiv

  12. [17]

    Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818,

    Xue, Z., Wu, J., Gao, Y ., Kong, F., Zhu, L., Chen, M., Liu, Z., Liu, W., Guo, Q., Huang, W., et al. Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818,

    Pith/arXiv arXiv

  13. [18]

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T

    URL https://arxiv.org/abs/2509.01563. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y ., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. InThirty- seventh Conference on Neural Information Processing Systems. Ye, Y ., He, X., Li, Z., Lin, B., Yuan, S., Yan, Z., Hou, B., and Yuan, L. Imgedit: A...

    Pith/arXiv arXiv

  14. [19]

    Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition

    Zhang, P., Dong, X., Wang, B., Cao, Y ., Xu, C., Ouyang, L., Zhao, Z., Duan, H., Zhang, S., Ding, S., et al. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309.15112,

    Pith/arXiv arXiv

  15. [20]

    Mm-rlhf: The next step forward in multimodal llm alignment

    Zhang, Y ., Yu, T., Tian, H., Fu, C., Li, P., Zeng, J., Xie, W., Shi, Y ., Zhang, H., Wu, J., et al. Mm-rlhf: The next step forward in multimodal llm alignment. InF orty-second International Conference on Machine Learning. Zhang, Y .-F., Lu, X., Hu, X., Fu, C., Wen, B., Zhang, T., Liu, C., Jiang, K., Chen, K., Tang, K., et al. R1-reward: 11 Joint Reward M...

    Pith/arXiv arXiv 1909

  16. [2017]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    Pith/arXiv arXiv

  17. [2022]

    Paco-rl: Advancing reinforcement learning for consistent image generation with pairwise reward modeling.arXiv preprint arXiv:2512.04784,

    10 Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models Ping, B., Jia, C., Luo, M., Xia, C., Shen, X., Dang, Z., and Qian, H. Paco-rl: Advancing reinforcement learning for consistent image generation with pairwise reward modeling.arXiv preprint arXiv:2512.04784,

    arXiv

  18. [2023]

    URL https: //arxiv.org/abs/2308.12966. Bai, S., Cai, Y ., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y ., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Re...

    Pith/arXiv arXiv

  19. [2024]

    F., Batifol, S., Blattmann, A., Boesel, F., Con- sul, S., Diagne, C., Dockhorn, T., English, J., English, Z., Esser, P., et al

    Labs, B. F., Batifol, S., Blattmann, A., Boesel, F., Con- sul, S., Diagne, C., Dockhorn, T., English, J., English, Z., Esser, P., et al. Flux. 1 kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506.15742,

    Pith/arXiv arXiv

  20. [2025]

    Christiano, P

    URLhttps://arxiv.org/abs/2511.21631. Christiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from hu- man preferences. InProceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4302–4310,

    Pith/arXiv arXiv