Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

RealMat: Realistic Materials with Diffusion and Reinforcement Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RealMat claims that finetuning SDXL on packed SVBRDF grids and then reinforcing a learned realism score makes text-to-material generation more realistic than prior methods and its own base model.

desk verdict RealMat is a solid, practical material generator whose RL-for-realism recipe is credible; the main weakness is that the realism reward serves as both optimizer and quantitative metric, though the user studies keep the central claim standing. read the letter →

arxiv 2509.01134 v1 pith:RF4EVFUC submitted 2025-09-01 cs.GR cs.CV

Xilong Zhou , Pedro Figueiredo , Miloš Hašan , Valentin Deschaintre , Paul Guerrero , Yiwei Hu , Nima Khademi Kalantari This is my paper
classification cs.GRcs.CV
keywords materialsSVBRDFgenerativemodelsdiffusionreinforcementlearningrealismrewardtext-to-materialgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to solve a known weakness of text-to-material generators: most are trained on synthetic data, so outputs look too clean and digital. RealMat keeps the realistic visual priors of a large text-to-image diffusion model while teaching it to output material maps. It does this in two stages: first finetune SDXL on synthetic SVBRDF maps packed into 2x2 grids; then use reinforcement learning to push outputs toward a realism reward trained on a large dataset of real material photographs. The paper reports that this second stage visibly improves realism, and that in a user study participants prefer RealMat materials over those of competing systems. If the approach is right, it offers a way to get realism without sacrificing the diversity and text control of large pretrained image models.

What carries the argument

The key mechanism is the 2x2 SVBRDF grid: albedo, height, roughness, and metallic maps arranged as quadrants of a single RGB image, so a pretrained text-to-image diffusion model (SDXL) can be finetuned to generate all four material channels at once while keeping its learned realism and diversity. The second load-bearing mechanism is the realism reward function, a linear layer on CLIP image features trained with MSE plus total-variation regularization on binary labels from filtered real photographs and generated materials; it is applied to a rendering of the generated grid under a random natural HDR environment. DDPO then treats the denoising trajectory as a Markov decision process and maximi

What would settle it

Take materials generated from prompts outside the 16 training categories, render them under novel lighting, and ask independent raters to compare pre- and post-RL outputs. If post-RL outputs are not preferred at a rate above chance even as the learned reward score rises, the reward is a proxy that does not track human realism. A direct measurement: compute the correlation between the reward's scores and human pairwise realism judgments on a held-out mix of real photographs and generated materials; the central claim requires a strong positive correlation.

Watch

Extended reading notes

Core claim

On its own terms, RealMat's discovery is that the realism gap from finetuning a general image diffusion model on synthetic material maps can be closed by a second reinforcement-learning finetuning stage that maximizes a learned realism score instead of a pixel loss. The authors pack albedo, height, roughness, and metallicity into one 2x2 grid, finetune SDXL on synthetic variations, then use DDPO to maximize a reward computed from CLIP features of the material rendered under natural HDR lighting. They report that this reward separates real from synthetic test images (normalized means 0.723 vs 0.324), and that RL improves realism both on training prompts and unseen prompts. In their user study

Load-bearing premise

The load-bearing premise is that the realism score used as the RL reward—a linear model on CLIP features, trained with binary labels handed down from classifiers trained on just 2,000 human-annotated images per dataset—matches what humans mean by 'realistic' across the whole range of materials the generator can produce.

Editorial extensions

If this is right

  • Materials are generated directly as SVBRDF maps from text, with no separate acquisition or decoder stage, so the typical baked-lighting artifacts of two-stage methods are avoided.
  • The realism gains carry over to prompts never used in RL training, so the reward does not merely memorize a small prompt set.
  • The realism score itself is a reusable material-realism estimator: the authors show it separates real photographs from synthetic renders, with normalized means of 0.723 versus 0.324.
  • Because RL is applied with LoRA on a frozen base, the recipe is relatively cheap and could be re-run as better base image generators appear.
  • The grid representation trades spatial resolution for four channels; if base resolution rises, the same packing can produce higher-quality material maps.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not pursue is to use the trained realism reward as a general evaluator for material assets, not only as an RL signal; it could filter synthetic datasets or rank generated outputs.
  • The same two-stage pattern of finetuning a pretrained image diffusion model on a packed multi-channel output and then reinforcing a learned realism score may transfer to other graphics outputs, such as normal maps, displacement maps, or procedural node graphs, where synthetic supervision is precise but visually sterile.
  • Because the reward is trained on CLIP features and binary labels derived from only 2,000 human annotations per dataset, it may track photographic plausibility rather than physical reflectance correctness; a testable check is whether RL-improved materials stay correct under novel lighting beyond the 200 HDR environments used.
  • Using live human preference feedback instead of classifier-generated labels could push the same framework further toward what users call realistic, and would also reveal whether the learned reward saturates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. RealMat proposes a two-stage text-to-material generator. Stage 1 fine-tunes Stable Diffusion XL on synthetic SVBRDF data arranged as 2×2 grids (albedo, height, roughness, metallic), inheriting SDXL's image priors. Stage 2 applies DDPO reinforcement learning with a learned realism reward: a linear layer on CLIP image features, trained with MSE loss on binary realism labels produced by linear CLIP classifiers trained on small human-annotated subsets of real photographs and generated materials. The paper evaluates with reward-score curves, ablations of prompt count and TV regularization, and two user studies comparing RealMat against its base model and against MatFuse, ReflectanceFusion, and PhotoMat.

Significance. If the results hold, RealMat offers a practical recipe for materially improving the realism of text-driven SVBRDF generation by combining a strong image prior with RL fine-tuning. The paper's strengths include the clean 2×2 grid formulation, the use of large-scale real photograph filtering, the DDPO-based RL stage, and two user studies with reported confidence intervals that independently ground the main claim that RealMat improves realism over its base model and related work. The computational setup is described in detail and the ablations cover relevant design choices. The main weakness is that the realism reward function—the key novelty—is not validated against held-out human judgments on generated materials, and the same reward is used both as the RL objective and as the primary quantitative evaluation metric, creating circularity for the non-user-study evidence.

major comments (3)
  1. [Sec. 4.1, 4.3, Eq. (5)–(6), Fig. 7, Sec. 6.3] The realism reward r_phi is a load-bearing component: it is maximized by DDPO in Stage 2 and then used as the metric in Fig. 7 and in the Sec. 6.3 ablations (including the choice of N=100 prompts). However, r_phi is trained on binary labels produced by linear CLIP classifiers that were themselves trained on only 2,000 manually annotated images per dataset, with empirically chosen thresholds 0.2/0.4, and no held-out human validation of r_phi is reported. Consequently, the quantitative evidence in Fig. 7 measures the quantity being optimized, not an independent realism judgment. The user studies in Sec. 6.2 and 6.4 break this circularity for the headline before/after and method-comparison claims, but the ablations and the contribution claim that the reward 'can accurately evaluate the realism of rendered materials' remain grounded in an unvalidated metric. The Limitations section (Sec. 7)
  2. [Sec. 6.1] The validation of the reward function shows only that average scores differ between a real test set (0.73/0.723) and a synthetic test set (0.516/0.324 after min-max normalization based on extrema 0.878/0.343). This demonstrates a real-vs-synthetic separation, but not that r_phi orders generated materials by human realism—the property that actually matters for the RL objective. The source of the normalization extrema is not stated. The abstract and contribution bullet claim a reward that 'can accurately evaluate realism,' which is stronger than the evidence presented. Either add a human-calibration experiment or revise the claim to 'separates real photographs from synthetic renders.'
  3. [Sec. 6.2, user study design] The RL fine-tuning user study compares materials before and after RL, showing a statistically significant but modest preference for the RL-tuned outputs (46.8% vs 28.9%, CIs [42.9%,50.7%] and [25.4%,32.4%]). However, the design lacks a control for additional training without the realism reward (e.g., DDPO with a constant or random reward, or continued SDXL fine-tuning). Without such a control, the improvement cannot be unambiguously attributed to the realism reward as opposed to the extra optimization steps or the LoRA update. This does not undermine the main comparison, but it weakens the claim that the reward function is the cause of the improvement.
minor comments (7)
  1. [Sec. 4.1] The thresholds 0.2/0.4 are described as 'empirically determined' but no evidence is given. Report classifier accuracy, precision/recall, or a sensitivity analysis to justify these values.
  2. [Sec. 2] Typo: 'Lee at al.' should be 'Lee et al.'
  3. [Eq. (5)] The notation 'L_tv(CLIP(I))' is ambiguous; the opening parenthesis after L_tv is not matched in the displayed equation. Also, specify the CLIP feature extraction step (e.g., which layer/patch pooling is used).
  4. [Fig. 7] The y-axis is 'Normalized realism score'; specify the normalization procedure and whether the scores come from the same reward model used for training (which would be circular for this plot as well).
  5. [Sec. 6.2, Fig. 10] The progressive fine-tuning snapshots in Fig. 10 are informative, but the selection of intermediate epochs is not described. State which epochs are shown and whether prompts/seeds are fixed.
  6. [Sec. 6.3] The claim that N=100 is optimal because it 'achieves the highest realism score based on our analysis' is based on Fig. 7(b), which appears to show overlapping curves. Report means/standard deviations over multiple seeds to support the choice.
  7. [Sec. 6.2] The study reports 32 sub-survey responses and 641 feedback points, but also 421 different material pairs; clarify how pairs are distributed across surveys and whether some pairs were repeated across participants.

Circularity Check

1 steps flagged · score 6.0 of 10

Quantitative realism-gain evidence reduces to the RL objective itself; user studies provide independent grounding.

  1. fitted input called prediction [Sec. 4.3 Eq. (6); Sec. 6.2 Fig. 3; Sec. 6.3 Fig. 7]
    "The accumulated reward R in Eq. 4 is then defined as: R(τ) ≔ rφ(g(d(x0), L)) ... We evaluate how the realism scores of generated materials evolve using both train and test prompts as our RL fine-tuning progresses in Fig. 7 (a)."

    The 'realism score' plotted in Fig. 7 and shown in Fig. 3 is exactly the reward rφ that DDPO maximizes (Eq. 4 with R from Eq. 6). Because RL training optimizes this reward, its increase during training is guaranteed by construction and cannot independently confirm improved realism. Moreover, rφ is itself a fitted linear layer trained with MSE (Eq. 5) to binary labels produced by the paper's own thresholded classifiers (Sec. 4.1), so the quantitative 'realism' gain is a fitted input renamed as an evaluation score. The independent user studies (Sec. 6.2, 6.4) partially break the circularity, but the numerical evidence in Fig. 3/Fig. 7 does not.

full rationale

The central claim that RealMat produces more realistic materials is not wholly circular: the RL finetuning user study (46.8% vs 28.9% preference) and the competing-methods user study (53% vs 40% vs ReflectanceFusion) are external human judgments and do not depend on the paper's reward. The method itself is built on external components (SDXL, DDPO, CLIP, and a synthetic dataset following Vecchio et al.). However, the paper repeatedly uses the learned realism reward as its own quantitative evidence of improvement: Eq. 6 defines the DDPO reward as rφ, and Fig. 7/Fig. 3 plot the same rφ as 'realism score' during and after the RL stage. Since DDPO maximizes that exact function, the measured increase is tautological with respect to the optimization objective. The reward is also a fitted model trained on thresholded labels from classifiers trained on only 2,000 human annotations per dataset; this does not make the user studies circular, but it means the numerical realism gains should not be treated as independent validation. The Limitations section does not flag this reward-as-metric loop. Overall: partial circularity in the quantitative evaluation, with the central claim rescued by independent user studies.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about the transferability of CLIP features to realism judgment and the validity of the reward as a training signal. The main free parameters are thresholds and loss weights, chosen without extensive ablation. No new physical entities are introduced.

free parameters (6)
  • realism label thresholds = 0.2 (Real Photographs), 0.4 (Generated Materials)
    Empirically determined thresholds in Sec. 4.1 for converting classifier scores to binary realism labels; no sensitivity analysis.
  • reward loss weights lambda_1, lambda_2 = 1 and 100
    Chosen in Sec. 5; no ablation shown for these weights.
  • LoRA rank = 4
    Chosen in Sec. 5 for memory/compute tradeoff; no ablation.
  • Number of RL training prompts = 100
    Selected in Sec. 6.3 based on Fig. 7(b); final setting.
  • Number of RL epochs = 110
    Chosen in Sec. 5; no ablation on epoch count.
  • Number of natural lighting environments = 200
    Sampled from Poly Haven in Sec. 4.3; no analysis of sensitivity to this count.
assumptions (4)
  • domain assumption SDXL's pretrained prior contributes realism and diversity to material generation
    Key premise of the two-stage approach, stated in Sec. 1 and Sec. 4.2.
  • domain assumption CLIP image features are sufficient to predict material realism
    The realism reward and classifiers are linear layers on CLIP features (Sec. 4.1, 4.3).
  • domain assumption Realism under natural lighting is a valid proxy for overall material quality
    The reward is evaluated on renders under natural HDR lighting; user studies use similar renders, but the reward itself is never validated directly against human labels.
  • domain assumption DDPO can optimize diffusion models with a non-differentiable reward without reward hacking
    Relies on Black et al. 2023; no analysis of reward hacking in this domain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RealMat: Realistic Materials with Diffusion and Reinforcement Learning." pith.science (2026). https://pith.science/paper/RF4EVFUC

@misc{pith2026250901134,
  author       = {Pith},
  title        = {Pith review of: RealMat: Realistic Materials with Diffusion and Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RF4EVFUC}},
  note         = {Machine review of arXiv:2509.01134}
}
abstract

Generative models for high-quality materials are particularly desirable to make 3D content authoring more accessible. However, the majority of material generation methods are trained on synthetic data. Synthetic data provides precise supervision for material maps, which is convenient but also tends to create a significant visual gap with real-world materials. Alternatively, recent work used a small dataset of real flash photographs to guarantee realism, however such data is limited in scale and diversity. To address these limitations, we propose RealMat, a diffusion-based material generator that leverages realistic priors, including a text-to-image model and a dataset of realistic material photos under natural lighting. In RealMat, we first finetune a pretrained Stable Diffusion XL (SDXL) with synthetic material maps arranged in $2 \times 2$ grids. This way, our model inherits some realism of SDXL while learning the data distribution of the synthetic material grids. Still, this creates a realism gap, with some generated materials appearing synthetic. We propose to further finetune our model through reinforcement learning (RL), encouraging the generation of realistic materials. We develop a realism reward function for any material image under natural lighting, by collecting a large-scale dataset of realistic material images. We show that this approach increases generated materials' realism compared to our base model and related work.

Figures

Figures reproduced from arXiv: 2509.01134 by the authors.

Figure 1
Figure 1. We propose RealMat, a diffusion-based realistic text-to-material generator. We first finetune a Stable Diffusion XL (SDXL) model pretrained on images [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A visual illustration of RealMat. (a) In the first stage, we finetune SDXL for text-to-material generation using synthetic SVBRDF maps arranged in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Our model inherits the diversity and details of SDXL, and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (5 more)
Figure 7
Figure 7. Figure 7: Plots of realism score during training: (a) Normalized realism score [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: We demonstrate the effect of TV regularization when training realism [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Comparison of sampled results of PhotoMat and RealMat (we only [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: This figure shows materials resulting from increasing levels of RL fine-tuning with realism reward, starting from no fine-tuning (left) to fine-tuning for [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Visual comparison between RealMat, ReflectanceFusion and MatFuse. The maps from ReflectanceFusion have light baking artifacts (marked by red [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Generative Texture Filtering

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    A two-stage fine-tuning strategy on pre-trained generative models enables effective texture filtering that outperforms prior methods on challenging cases.

Reference graph

Works this paper leans on

5 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [3]

    IEEE Transactions on Visualization and Computer Graphics (2024)

    Single-Image SVBRDF Estimation Using Auxiliary Renderings as Intermediate Targets. IEEE Transactions on Visualization and Computer Graphics (2024). Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv pr...

    arXiv 2024

  2. [4]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684–10695. Sam Sartor and Pieter Peers. 2023. Matfusion: a generative diffusion model for svbrdf capture. In SIGGRAPH Asia 2023 Conference Papers . 1–10. Christoph Schuhmann, Romain Beaumont, Richard Vencu,...

    arXiv 2023

  3. [2022]

    Com- puter Graphics Forum 41, 2 (2022), 163–177

    MaterIA: Single Image High-Resolution Material Capture in the Wild. Com- puter Graphics Forum 41, 2 (2022), 163–177. https://doi.org/10.1111/cgf.14466 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14466 Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. 2003. A Data-Driven Reflectance Model. ACM Trans. Graph. 22, 3 (2003), ...

    work page doi:10.1111/cgf.14466 2022

  4. [2023]

    In ACM SIGGRAPH 2023 Conference Proceedings

    Deep SVBRDF Estimation from Single Image under Learned Planar Lighting. In ACM SIGGRAPH 2023 Conference Proceedings . 1–11. Xilong Zhou, Milos Hasan, Valentin Deschaintre, Paul Guerrero, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Nima Khademi Kalantari. 2023a. Photomat: A material generator learned from single flash photos. In ACM SIGGRAPH 2023 Confere...

    work page arXiv 2023

  5. [2024]

    https://polyhaven.com/hdris

    PolyHaven HDR environments. https://polyhaven.com/hdris. Adobe. 2023. Substance. https://substance3d.adobe.com/assets. Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. 2023. Training diffusion models with reinforcement learning.arXiv preprint arXiv:2305.13301(2023). Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettak...

    arXiv 2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.