Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

A Controllable Appearance Representation for Flexible Transfer and Editing

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that the appearance of a homogeneous object in a single image can be compressed without labels into a six-dimensional code — two hue axes, two light-direction axes, lightness, and gloss — that then conditions a diffusion…

desk verdict Solid self-supervised 6D appearance space with honest caveats, but geometry-independence is only tested on simple shapes and the diffusion part is proof-of-concept. read the letter →

arxiv 2504.15028 v2 pith:OCPUQ3PJ submitted 2025-04-21 cs.GR cs.CV

Santiago Jimenez-Navarro , Julia Guerrero-Viu , Belen Masia This is my paper · ORCID
classification cs.GRcs.CV
keywords materialappearancedisentangledrepresentationlearningself-supervisedlatentspaceinterpretabilitytransfereditingFactorVAEcompact
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the visual appearance of a homogeneous, opaque object in a single image can be compressed, without any human labels, into a six-dimensional code in which two dimensions encode hue, two encode illumination direction, one encodes lightness, and one encodes gloss. The authors adapt a FactorVAE: they feed normal maps into the decoder so geometry is supplied rather than encoded, and they reshape the KL regularizer with an order-3 norm and an annealed weight to stop posterior collapse. They train on a self-built set of 98,550 synthetic renders combining 30 geometries, 365 measured BRDFs, and nine lighting conditions, then use the frozen encoder to condition a pretrained latent diffusion model through a lightweight cross-attention adapter. If the claim is right, ordinary images become editable in material terms, and appearance can be transferred from one or several exemplars with slider-like control over individual attributes.

What carries the argument

The load-bearing device is the six-dimensional latent vector $f\in\mathbb{R}^6$ produced by an adapted FactorVAE. The encoder maps an input image of a homogeneous object to this vector, while the decoder concatenates a normal map of the target geometry into its layers, so geometry is handed to the model rather than stored in the code. The training loss combines a smooth-L1 reconstruction term, a dimension-wise KL regularization term $\beta\,D_{KL}(q_\theta(z|x),p(z),n)$ with an order-$n$ norm and annealed weight, and a total-correlation term computed by a discriminator to encourage independent dimensions. A dimensionality sweep from 3 to 10 justifies the choice of 6, balancing the Mutual Information Ratio (interpretability) and Mutual Information Score (disentanglement). In the diffusion stage, the frozen encoder becomes the appearance-conditioning branch, with a lightweight cross-attention adapter translating $f$ into generation guidance and an edge-plus-depth conditioning branch supplying the target geometry.

What would settle it

A concrete check is to render the same measured material and lighting on several complex unseen shapes, encode each image, and compare the six-dimensional codes: if codes drift with geometry, or if stepping along the gloss or hue dimension changes shadows or shape-dependent highlights, then geometry has leaked into the appearance code. The supplement already reports a template for this, with mean cosine similarity 0.879 between same-material pairs on two different geometries versus 0.321 for different materials; repeating that measurement on complex geometries with a full latent-traversal inspection would settle the claim.

Watch

Extended reading notes

Core claim

The central claim, stated by the authors, is that a variational autoencoder with a total-correlation disentanglement objective can learn an interpretable and disentangled image-space code for material appearance without explicit supervision, provided the decoder is given the target geometry as normal maps. The learned six dimensions turn out to be: lightness; two hue axes that the model aligns approximately perpendicular in the chromatic circle; two light-direction axes; and gloss. This code lives in a 6D space selected by sweeping dimensionality from three to ten and balancing interpretability and disentanglement metrics. The paper further shows that the same encoder can drive a diffusion-based inpainting pipeline, enabling appearance transfer, selective transfer of chosen dimensions from different images, interpolation between materials, and fine-grained editing of individual attributes in real photographs, even though training is synthetic. The authors acknowledge limits: the autoencoder alone degrades on geometries very different from training, and the representation covers only homogeneous opaque materials under moderate lighting.

Load-bearing premise

The load-bearing premise is that feeding the decoder the target geometry as normal maps keeps geometry out of the six-dimensional code, a premise that is only measured on simple blob and sphere test geometries (Section 4.1) while the paper itself reports artifacts when decoding very different geometries (Section 4.2, supplement S3.3).

Editorial extensions

If this is right

  • A single photograph of a homogeneous object can be compressed to a 6D vector carrying material and lighting, so appearance transfer no longer needs a text prompt or human-annotated labels.
  • Because the dimensions are independent, users can move one attribute at a time — gloss, hue, lightness, or light direction — while the other attributes stay fixed.
  • Appearance can be assembled from several exemplars by taking different dimensions from different images, such as hue from one object and gloss and lightness from another.
  • The image-space code transfers to real photographs even though training is synthetic, letting a real object's appearance be re-rendered on a new geometry.
  • Edits remain local to the material and leave geometry and background intact, which is the advantage the paper claims over text-instruction editing methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the code is as geometry-free as claimed, the same encoder could serve as a retrieval descriptor for material search across arbitrary object shapes; the paper mentions retrieval only as future work, so this is an extension rather than a paper claim.
  • The same self-supervised recipe, applied to render sets that vary texture, roughness, anisotropy, or translucency, could plausibly yield interpretable dimensions for those attributes; the paper does not test this.
  • Because only two dimensions encode illumination direction, selective transfer between images with different environment colors or high-frequency lighting structures will likely fail; the paper itself reports degradation on high-frequency lighting, so the practical scope of the code is limited to moderate, roughly neutral lighting.
  • The reported preference for the proposed transfer over a semantic-image-embedding baseline (61.5% versus 38.5% in a user study) suggests geometry leakage in the baseline is a real practical issue, but a larger study that varies geometry complexity would be needed to know how much the advantage generalizes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a self-supervised method for learning a compact, interpretable appearance representation from images of homogeneous, opaque objects. A FactorVAE-based encoder maps an input image to a six-dimensional latent code, while the decoder is conditioned on normal maps to encourage the code to capture material and illumination rather than geometry. The loss is modified with a norm-based KL regularizer and beta-annealing to mitigate posterior collapse. The authors introduce a synthetic dataset of 98,550 renders (30 geometries, 365 measured BRDFs, 9 lighting conditions) and report quantitative disentanglement, interpretability, and reconstruction comparisons against betaVAE, betaTCVAE, and FactorVAE. They then use the frozen encoder to condition a diffusion pipeline via a lightweight IP-Adapter, enabling appearance transfer, selective attribute transfer, editing, and interpolation. Code, models, and dataset are released.

Significance. If the central claim holds, this is a useful contribution: a six-dimensional, self-supervised appearance code that separates hue, gloss, lightness, and illumination would enable fine-grained control in material transfer and editing without human labels. The paper's strengths include the public release of code, models, and dataset; the use of measured BRDFs; systematic ablations of the loss modifications; and quantitative comparisons to three VAE baselines. The compactness of the representation and the demonstration that it can condition a diffusion pipeline are also valuable. However, the geometry-independence claim--which is load-bearing for selective transfer--is not directly measured on complex geometry, and the application-level evaluation is mostly qualitative. These issues are addressable with additional experiments, so the contribution is defensible but needs revision.

major comments (4)
  1. [Sec. 4.1, Table 1; Sec. 5.2] The headline disentanglement and interpretability metrics are computed only on a test subset deliberately limited to blob and sphere geometries (Sec. 4.1). Complex geometry is used only for reconstruction quality, not for disentanglement or interpretability. This leaves the central claim that each latent dimension varies only with material/illumination and not with geometry untested exactly where it is hardest. Supplement S3.4 reports an aggregate cosine similarity of 0.879 between embeddings of the same material rendered on cylinder and statuette, but an aggregate value does not rule out a single latent dimension carrying geometry-dependent signal. Because Section 5's selective transfer recombines individual dimensions across images, any per-dimension geometry leakage would directly corrupt the application. Please provide per-dimension invariance statistics (e.g., per-dimension variance or consistency across geometry renderings) on complex unseen geometries, not only an aggregate similarity.
  2. [Sec. 3.1.2, Table 2] Conditioning the decoder on normal maps is presented as 'compelling' the latent space to focus on appearance, but it is a soft inductive bias rather than a constraint, and no direct measurement shows that geometry information is absent from the code. The 'Without Normals' ablation in Table 2 shows changes in PSNR and MIR, but it does not measure what geometry-dependent information remains in the six dimensions. I recommend a direct leakage test: train a simple linear classifier on the latent codes to predict geometry identity (with appearance and illumination held fixed), or compute per-dimension classification accuracy / mutual information with geometry. Without such a test, the architecture claim in Sec. 3.1.2 is supported only by indirect evidence.
  3. [Sec. 4.1] The test set overlaps with the training set in materials: 'some materials are both in our training and test sets' (Sec. 4.1). Since the interpretability metrics (Z-min, MIR) use ground-truth labels on exactly this test set, the model could partially memorize material identity rather than learning general attribute semantics, inflating the reported interpretability scores. The authors note that illuminations and scene configurations differ, which mitigates but does not eliminate the confound. I ask for a hold-out material split, or at least a breakdown of MIR/Z-min separately for materials seen and unseen during training, so readers can gauge generalization to genuinely new materials.
  4. [Sec. 5.2, S6.2] The diffusion-pipeline evaluation for appearance transfer and editing is largely qualitative. The only quantitative evidence is a two-alternative user study with 20 participants (61.5% vs. 38.5% preference over ZeST), reported in S6.2 without a significance test. Moreover, there is no measurement of whether traversing one latent dimension in the diffusion outputs changes only the intended attribute and leaves others unchanged. Given that the abstract and Section 5 claim 'fine-grained control' and 'strong disentanglement,' I recommend adding a quantitative or at least statistically sound evaluation of the diffusion outputs, for example an attribute-classification test on generated images or a significance test for the user preference data.
minor comments (5)
  1. [Eq. (1)] The definition of the norm-based KL term is difficult to parse as typeset: D_KL(Q,P,n) appears to be a scalar expression without an explicit index over latent dimensions, so the 'norm of order n' is not reproducible from the equation. Please give the explicit form, e.g., (sum_j KL_j^n)^{1/n} or the equivalent used in the implementation.
  2. [Table 1] The GTC column shows '-' for betaVAE and betaTCVAE. If GTC was not computed for these baselines, state why; otherwise provide the values for completeness.
  3. [Sec. 4.3, Fig. 6] The KL evolution plots are described as showing how much information each dimension stores. Since the y-axis is not labeled in the figure, please add a caption note that higher values correspond to larger KL distance from the standard normal prior, and clarify the scale used.
  4. [Sec. 4.2, Fig. 3] The semantic names of the dimensions (lightness, hue #1, hue #2, light direction #1/#2, gloss) are assigned after training. This is acceptable, but the figure caption should make explicit that these names are post hoc interpretations of traversal behavior, not learned labels.
  5. [S6.1] The diffusion-based posterior traversals omit the two illumination dimensions. This is a reasonable choice, but it should be stated in the main text or figure caption so readers do not infer that illumination traversals were validated in the diffusion pipeline.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 6D latent code is trained without appearance labels, and the semantic attribute names are assigned and evaluated post hoc against external labels and traversals.

full rationale

The paper's central derivation is self-contained. The 6D latent representation is trained by minimizing Eq. 1, which combines a reconstruction term, a per-dimension KL regularizer, and a total-correlation term; no appearance labels enter the loss, and the semantic names (lightness, hue, light direction, gloss) are assigned a posteriori from traversal visualizations (Sec. 4.2, Fig. 3). Interpretability metrics Z-min and MIR are computed after training using ground-truth labels from the test dataset (Sec. 4.1); these labels are used for evaluation only and are not fitted parameters renamed as predictions. The choice of six latent dimensions is presented as a design decision analyzed in Sec. 4.3, not as a quantity predicted by the method. The normal-map conditioning of Sec. 3.1.2 is an inductive bias that encourages geometry to be handled by the decoder; it is a soft architectural choice, not a circular constraint, and the paper candidly documents its limitations, including failure on unseen geometries (S3.3), out-of-distribution illumination (Fig. 12), lack of perceptual linearity, and the absence of hard disentanglement constraints. The one citation to prior work by a coauthor, the Serrano dataset [SCW*21], supplies test images, measured BRDFs, and ground-truth labels as external published data; it is not invoked as a uniqueness theorem, nor does it justify the loss or the claimed disentanglement. The restriction of the headline disentanglement metrics to blob/sphere test geometries weakens the evidence for geometry-invariance on complex shapes, but that is a validity and generalization concern, not a circularity: no equation or construction reduces the claimed result to its inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central representation claim rests on an assumed factorization of appearance into six independent latent dimensions, on normal-map conditioning removing geometry from the code, and on the synthetic dataset covering real-world homogeneous materials. The main free parameters are the latent dimensionality, the loss hyperparameters (beta, gamma, n), and the ControlNet weights. No new physical entities are introduced; the 6D latent is a learned representation, not a postulated physical quantity.

free parameters (5)
  • Latent dimensionality = 6
    Selected by sweeping dimensions 3 to 10 and choosing the best MIR/MIS balance (Fig. 5); a tuning choice, not a derived quantity.
  • Maximum KL weight beta = 2
    Beta in Eq. 1 is annealed linearly from 0 to 2 over the first 1000 epochs; ablation (b) shows beta=1 lowers MIR, so this value is a hyperparameter selected for the final model.
  • Total correlation weight gamma = 6
    Fixed gamma in Eq. 1, reported in S1.1 without a shown grid search.
  • KL norm order n = 3
    Norm order in the modified DKL term of Eq. 1; n=1 (summation) is ablated as worse in Table 2.
  • ControlNet weights = D=0.2, C=0.9
    Depth and Canny ControlNet weights chosen from a qualitative ablation in S6.3; needed for the diffusion transfer pipeline.
assumptions (5)
  • domain assumption The appearance of a homogeneous opaque object in an image is factorizable into six independent latent dimensions under a standard normal prior.
    The FactorVAE objective and the 6D latent size assume this factorization exists and is learnable from the rendered dataset (Sec. 3.1, Fig. 5).
  • domain assumption Feeding normal maps to the decoder removes geometry as a latent factor of variation.
    Sec. 3.1.2 states this, but the paper does not directly measure geometry leakage in the latent code.
  • domain assumption The synthetic dataset of 365 measured BRDFs, 30 geometries, and 9 rotations of one environment map is representative enough for real-world homogeneous materials.
    Sec. 3.2; the paper limits its claim to homogeneous opaque materials and moderate lighting, and Fig. 12 shows failure on high-frequency illumination.
  • domain assumption Ground-truth attribute labels in the Serrano test set are a valid measure of interpretability.
    MIR and Z-min use these labels (Sec. 4.1); if the labels do not match human-perceived attributes, interpretability scores lose meaning.
  • domain assumption Pretrained generative components (RealisticVisionXL4.0, ControlNets, depth estimator) behave consistently when conditioned from a 6D latent via IP-Adapter.
    The Sec. 5.1 pipeline relies on frozen pretrained models; there is no guarantee that their conditioning preserves the latent space's disentanglement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Controllable Appearance Representation for Flexible Transfer and Editing." pith.science (2026). https://pith.science/paper/OCPUQ3PJ

@misc{pith2026250415028,
  author       = {Pith},
  title        = {Pith review of: A Controllable Appearance Representation for Flexible Transfer and Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OCPUQ3PJ}},
  note         = {Machine review of arXiv:2504.15028}
}
read the original abstract

We present a method that computes an interpretable representation of material appearance within a highly compact, disentangled latent space. This representation is learned in a self-supervised fashion using an adapted FactorVAE. We train our model with a carefully designed unlabeled dataset, avoiding possible biases induced by human-generated labels. Our model demonstrates strong disentanglement and interpretability by effectively encoding material appearance and illumination, despite the absence of explicit supervision. Then, we use our representation as guidance for training a lightweight IP-Adapter to condition a diffusion pipeline that transfers the appearance of one or more images onto a target geometry, and allows the user to further edit the resulting appearance. Our approach offers fine-grained control over the generated results: thanks to the well-structured compact latent space, users can intuitively manipulate attributes such as hue or glossiness in image space to achieve the desired final appearance.

Figures

Figures reproduced from arXiv: 2504.15028 by the authors.

Figure 1
Figure 1. Our model learns a disentangled and interpretable latent space of appearance in a self-supervised manner, without human￾annotated data. Given an input image depicting a homogeneous object (left), it can be encoded into this space, which can then be traversed to generate meaningful variations of appearance (here, traversals along two dimensions encoding hue and gloss are shown). This encoded representation of appeara… view at source ↗
Figure 2
Figure 2. Diagram of our VAE-based architecture. The encoder creates a low-dimensional, disentangled representation f of the ap￾pearance of the input image. The decoder learns to apply the in￾coming appearance to a reference geometry, specified with a nor￾mal map, which is concatenated in the decoder pipeline. The red box illustrates the discriminator used to compute the TC term (see text for details). 3.1.1. Avoiding Posteri… view at source ↗
Figure 3
Figure 3. Prior traversals sampling our 6D latent space. Each row samples a different dimension of the space, starting from a neutral, zero-valued feature vector (central column). The feature vector is fed to the decoder together with the normal map of the Havran geometry to generate the images shown. Our unsupervised model yields dimensions that are not only disentangled, but also in￾terpretable, as indicated by the attribut… view at source ↗
Figures from the paper (29 more)
Figure 5
Figure 5. Figure 5: Latent space dimensionality analysis. Evolution of met￾rics for interpretability (MIR, higher is better) and disentanglement (MIS, lower is better) for models whose latent space dimensional￾ity ranges between 3 and 10. Our 6-dimensional space achieves the best balance …
Figure 6
Figure 6. Figure 6: Prior traversals and KL evolution plots of each model trained with latent space dimensionalities ranging from 3 to 10. KL evolution plots shows the dimension-wise evolution, during training, of the KL distance between the learned distributions and the standard normal d…
Figure 7
Figure 7. Figure 7: Diffusion-based pipeline for proof-of-concept applications of our space. Our proposed pipeline uses two branches to condition the diffusion-based generative process with Stable Diffusion XL (SDXL). The appearance conditioning branch leverages our encoder to produce a 6…
Figure 8
Figure 8. Figure 8: Appearance transfer and editing results of our pipeline for real-world images leveraging our disentangled appearance repre￾sentation. In each example, we use two input images, with the leftmost one as the target. We selectively transfer material and/or illumination pro…
Figure 9
Figure 9. Figure 9: Interpolation between two materials in our latent space. The two ends of the progression show the result of transferring the material of two real-world objects, a blue glossy spoon and a pink rough llama, to a target geometry. All results follow the illumination of the…
Figure 10
Figure 10. Figure 10: Examples highlighting the disentanglement issue in ZeST [CSM*24] for appearance transfer. The appearance refer￾ence image of the first row is a rendering of a homogeneous bud￾dha, while the second row uses a real photograph of the Statue of Liberty. In contrast to our…
Figure 11
Figure 11. Figure 11: Comparison of the appearance editing feature of our pipeline with two state-of-the-art solutions. Using a reference image (left), we show the progression of sequentially editing two attributes (right). The disentanglement of our latent space and the design of the diff…
Figure 12
Figure 12. Figure 12: Limitations. Examples of the behavior of our autoen￾coder when reconstructing out-of-distribution appearances. The sphere and blob samples are illuminated with high-frequency light￾ing. Reconstructing the Havran geometry using their respective em￾beddings, the model s…
Figure 13
Figure 13. Figure 13: Grid plots, showing the appearance reconstructed by the decoder when lineally combining pairs of dimensions of the latent space. We show: (left) hue #1 and hue #2, (center) light dir. #2 and gloss, and (right) light dir. #2 and lightness. Interestingly, the model has …
Figure 14
Figure 14. Figure 14: Additional results of the posterior traversal plot. We use as reference three unseen samples, reconstructing the Havran geometry. ods, we see how combining the identified dimensions results in progressive changes in the appearance of the Havran geometry. S3.2. Additio…
Figure 16
Figure 16. Figure 16: Ablations. Evolution of different alternative reconstruc￾tion losses during training. Note that y-axis is log-scale. interpretability, measured by MIR, in [PITH_FULL_IMAGE:figures/full_fig_p017_16.png]
Figure 17
Figure 17. Figure 17: Ablation study on the two alternatives for downsam￾pling the normal map information, namely nearest neighbors (NN), and bilinear. Left: the reference geometry (cat), with its respective resulting appearance after applying each of the two algorithms. Right: samples use…
Figure 19
Figure 19. Figure 19: Ablation on the influence of the ControlNet weights over the final result. On top are represented the geometry used as reference, with its respective estimated depth and Canny maps. Each row contains the result of running the inference pipeline with the same custom re…
Figure 20
Figure 20. Figure 20: Representative samples of the custom training dataset. Rows one, two, and three contain samples of the 30 geometries used. Rows four, five, and six contain a representative set of the 365 measured materials. Row seven shows the nine illuminations used in the dataset …
Figure 21
Figure 21. Figure 21: Posterior traversals plot generated using car and blob as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: Posterior traversals plot generated using car and bunny as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Posterior traversals plot generated using car and chair as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]
Figure 24
Figure 24. Figure 24: Posterior traversals plot generated using car and jar as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p022_24.png]
Figure 25
Figure 25. Figure 25: Posterior traversals plot generated using car and teapot as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p023_25.png]
Figure 26
Figure 26. Figure 26: Posterior traversals plot generated using cat and blob as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p023_26.png]
Figure 27
Figure 27. Figure 27: Posterior traversals plot generated using cat and bunny as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p024_27.png]
Figure 28
Figure 28. Figure 28: Posterior traversals plot generated using cat and chair as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p024_28.png]
Figure 29
Figure 29. Figure 29: Posterior traversals plot generated using cat and jar as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p025_29.png]
Figure 30
Figure 30. Figure 30: Posterior traversals plot generated using cat and teapot as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p025_30.png]
Figure 31
Figure 31. Figure 31: Posterior traversals plot generated using napoleon and blob as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p026_31.png]
Figure 32
Figure 32. Figure 32: Posterior traversals plot generated using napoleon and bunny as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p026_32.png]
Figure 33
Figure 33. Figure 33: Posterior traversals plot generated using napoleon and chair as geometry and appearance references. Transfer marked in green. Geometry & Illumination Appearance Lightness Hue #1 Hue #2 Gloss [PITH_FULL_IMAGE:figures/full_fig_p027_33.png]
Figure 34
Figure 34. Figure 34: Posterior traversals plot generated using napoleon and jar as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p027_34.png]
Figure 35
Figure 35. Figure 35: Posterior traversals plot generated using napoleon and teapot as geometry and appearance references. Transfer marked in green [PITH_FULL_IMAGE:figures/full_fig_p028_35.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Hybrid Neural-Microfacet BRDF Model for Real-Time Rendering

    cs.GR 2026-08 conditional novelty 6.0 of 10

    A hybrid BRDF model, combining a GGX analytical term with a tiny learned residual and gating network, fits measured materials more accurately than fully neural models at equal memory cost.

Reference graph

Works this paper leans on

66 extracted references · 64 canonical work pages · cited by 1 Pith paper

  1. [1]

    Improv- ing image generation with better captions

    [BGJ*23] B ETKER , J AMES , G OH, G ABRIEL , J ING, L I, et al. “Improv- ing image generation with better captions”. Computer Science. 2.3 (2023), 8

    work page 2023

  2. [2]

    An Adaptive Pa- rameterization for Efficient Material Acquisition and Rendering

    [DJ18] D UPUY , J ONATHAN and J AKOB , W ENZEL . “An Adaptive Pa- rameterization for Efficient Material Acquisition and Rendering”. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia)37.6 (Nov. 2018), 274:1–274:18 2, 4,

    work page 2018

  3. [3]

    Instructpix2pix: Learning to follow image editing instruc- tions

    [BHE23] B ROOKS , T IM, H OLYNSKI , A LEKSANDER , and E FROS , ALEXEI A. “Instructpix2pix: Learning to follow image editing instruc- tions”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 18392–18402 2, 10,

    work page 2023

  4. [4]

    Colorpeel: Color prompt learning with diffusion models via color and shape disentangle- ment

    [BWVvdW24] B UTT, M UHAMMAD ATIF, W ANG, K AI, V AZQUEZ - CORRAL , J AVIER, and van de W EIJER , J OOST . “Colorpeel: Color prompt learning with diffusion models via color and shape disentangle- ment”. Proceedings of the European Conference on Computer Vision . Springer. 2024, 456–472

    work page 2024

  5. [5]

    Infogan: Interpretable representation learning by information maximizing gener- ative adversarial nets

    [CDH*16] C HEN, X I, D UAN, YAN, H OUTHOOFT , R EIN, et al. “Infogan: Interpretable representation learning by information maximizing gener- ative adversarial nets”. Advances in Neural Information Processing Sys- tems 29 (2016)

    work page 2016

  6. [6]

    Isolating sources of disentanglement in varia- tional autoencoders

    [CLGD18] C HEN, R ICKY TQ, L I, X UECHEN , G ROSSE , R OGER B, and DUVENAUD , DAVID K. “Isolating sources of disentanglement in varia- tional autoencoders”. Advances in Neural Information Processing Sys- tems 31 (2018) 3,

    work page 2018

  7. [7]

    Zest: Zero-shot material transfer from a single image

    [CSM*24] C HENG , T A-Y ING, S HARMA , P RAFULL , M ARKHAM , A N- DREW , et al. “Zest: Zero-shot material transfer from a single image”. Proceedings of the European Conference on Computer Vision. Springer. 2024, 370–386 2, 3, 8, 10,

    work page 2024

  8. [8]

    Single-image svbrdf capture with a rendering-aware deep network

    [DAD*18] D ESCHAINTRE , V ALENTIN , A ITTALA , M IIKA , D URAND , FREDO , et al. “Single-image svbrdf capture with a rendering-aware deep network”. ACM Transactions on Graphics (TOG)37.4 (2018), 1–15

    work page 2018

Show all 66 references
  1. [10]

    Unsupervised learning predicts human percep- tion and misperception of gloss

    [SAF21] S TORRS , K ATHERINE R, A NDERSON , B ARTON L, and F LEM - ING, R OLAND W. “Unsupervised learning predicts human percep- tion and misperception of gloss”. Nature Human Behaviour 5.10 (2021), 1402–1417 3,

    2021

  2. [11]

    2022, 453–464 2–4,

    Wiley Online Library. 2022, 453–464 2–4,

    2022

  3. [13]

    Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

    [FLL*19] F U, H AO, L I, C HUNYUAN , L IU, X IAODONG , et al. “Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing”. NAACL. 2019

    2019

  4. [14]

    2014, 91–99

    Wiley On- line Library. 2014, 91–99

    2014

  5. [15]

    Disentangling VAE

    [DKLM19] D UBOIS , YANN, K ASTANOS , A LEXANDROS , L INES , DAVE, and M ELMAN , B ART. Disentangling VAE. http://github.com/ YannDubs/disentangling-vae/. Mar. 2019

    2019

  6. [16]

    Texsliders: Diffusion-based texture editing in clip space

    [GHR*24] G UERRERO -VIU, J ULIA , H ASAN , M ILOS , R OULLIER , ARTHUR , et al. “Texsliders: Diffusion-based texture editing in clip space”. ACM SIGGRAPH 2024 Conference Papers. 2024, 1–11

    2024

  7. [17]

    Real-world illumination and the perception of surface re- flectance properties

    [FDA03] F LEMING , R OLAND W., D ROR, R ON O., and A DELSON , E D- WARD H. “Real-world illumination and the perception of surface re- flectance properties”. Journal of Visual Communication and Image Rep- resentation 3.5 (2003). ISSN : 1534-7362

    2003

  8. [18]

    Interpretable Disentangled Parametrization of Measured BRDF with β-V AE

    [BSP22] B ENAMIRA , A LEXIS , S HAH, S ACHIN , and P ATTANAIK , SUMANTA . “Interpretable Disentangled Parametrization of Measured BRDF with β-V AE”.arXiv preprint arXiv:2208.03914 (2022)

    2022 arXiv

  9. [19]

    Fast R-CNN

    [Gir15] G IRSHICK , R OSS. “Fast R-CNN”. International Conference on Computer Vision (ICCV). 2015

    2015

  10. [20]

    2024, e15037

    Wiley Online Library. 2024, e15037

    2024

  11. [21]

    2016, 1–12

    Wiley Online Library. 2016, 1–12

    2016

  12. [22]

    Deep- BRDF: A Deep Representation for Manipulating Measured BRDF

    [HGC*20] H U, B INGYANG , G UO, J IE, C HEN, Y ANJUN , et al. “Deep- BRDF: A Deep Representation for Manipulating Measured BRDF”. Computer Graphics Forum 39.2 (2020), 157–166 2,

    2020

  13. [23]

    Denoising diffusion probabilistic models

    [HJA20] H O, JONATHAN , JAIN, AJAY, and ABBEEL , PIETER . “Denoising diffusion probabilistic models”. Advances in Neural Information Pro- cessing Systems 33 (2020), 6840–6851

    2020

  14. [24]

    beta- vae: Learning basic visual concepts with a constrained variational frame- work

    [HMP*17] H IGGINS , I RINA , M ATTHEY , L OIC, PAL, A RKA, et al. “beta- vae: Learning basic visual concepts with a constrained variational frame- work.” ICLR (Poster) 3 (2017) 3,

    2017

  15. [25]

    LoRA: Low-Rank Adaptation of Large Language Models

    [HSW*22] H U, E DWARD J, S HEN, Y ELONG , W ALLIS , P HILLIP , et al. “LoRA: Low-Rank Adaptation of Large Language Models”. The Tenth International Conference on Learning Representations, ICLR. 2022

    2022

  16. [26]

    Mitsuba renderer

    [Jak10] J AKOB , W ENZEL . Mitsuba renderer . http://www.mitsuba- renderer.org. 2010 5,

    2010

  17. [27]

    A style- based generator architecture for generative adversarial networks

    [KLA19] K ARRAS , T ERO, L AINE , S AMULI , and A ILA, T IMO. “A style- based generator architecture for generative adversarial networks”. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 4401–4410

    2019

  18. [28]

    Disentangling by fac- torising

    [KM18] K IM, H YUNJIK and M NIH, A NDRIY . “Disentangling by fac- torising”. International Conference on Machine Learning . PMLR. 2018, 2649–2658 2–5. [KOF*23] K INOSHITA , YURI, OONO, KENTA, FUKUMIZU , KENJI , et al. “Controlling posterior collapse by an inverse Lipschitz con...

    2018

  19. [29]

    Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

    [KOH*24] K E, BINGXIN , OBUKHOV , ANTON , HUANG , SHENGYU , et al. “Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation”. Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition. 2024

    2024

  20. [30]

    Neumip: Multi-resolution neural materials

    [Kuz21] K UZNETSOV , A LEXANDR . “Neumip: Multi-resolution neural materials”. ACM Transactions on Graphics (TOG)40.4 (2021)

    2021

  21. [31]

    Deep convolutional inverse graphics network

    [KWKT15] K ULKARNI , T EJAS D, W HITNEY , W ILLIAM F, K OHLI , PUSHMEET , and T ENENBAUM , J OSH. “Deep convolutional inverse graphics network”. Advances in Neural Information Processing Systems 28 (2015)

    2015

  22. [32]

    [Lab23] L ABS, B LACK FOREST . FLUX. https : / / github . com / black-forest-labs/flux . 2023

    2023

  23. [33]

    Challenging common assumptions in the unsupervised learning of disentangled representations

    [LBL*19] L OCATELLO , FRANCESCO , BAUER , STEFAN , LUCIC , MARIO , et al. “Challenging common assumptions in the unsupervised learning of disentangled representations”. International Conference on Machine Learning. PMLR. 2019, 4114–4124

    2019

  24. [34]

    A Similarity Measure for Material Appearance

    [LMS*19] L AGUNAS , M ANUEL , M ALPICA , S ANDRA , S ERRANO , A NA, et al. “A Similarity Measure for Material Appearance”. ACM Transac- tions on Graphics (TOG) 38.4 (2019)

    2019

  25. [35]

    The joint role of geometry and illumination on material recognition

    S. Jimenez-Navarro, J. Guerrero-Viu & B. Masia / A Controllable Appearance Representation for Flexible Transfer and Editing 13 of 28 [LSGM21] L AGUNAS , MANUEL , SERRANO , ANA, GUTIERREZ , DIEGO , and M ASIA , B ELEN . “The joint role of geometry and illumination on material r...

    2021

  26. [36]

    Unsupervised learning reveals interpretable latent representations for translucency perception

    [LSX23] L IAO, C HENXI , S AWAYAMA, M ASATAKA , and X IAO, B EI. “Unsupervised learning reveals interpretable latent representations for translucency perception”. PLOS Computational Biology 19.2 (2023), e1010878

    2023

  27. [37]

    Understanding Posterior Collapse in Gener- ative Latent Variable Models

    [LTGN19] L UCAS , J AMES , T UCKER , G EORGE , G ROSSE , R OGER , and NOROUZI , M OHAMMAD . Understanding Posterior Collapse in Gener- ative Latent Variable Models. 2019

    2019

  28. [38]

    A Data-Driven Reflectance Model

    [MPBM03] M ATUSIK , W OJCIECH , P FISTER , H ANSPETER , B RAND , MATT, and M CMILLAN , L EONARD . “A Data-Driven Reflectance Model”. ACM Transactions on Graphics (TOG) 22.3 (July 2003), 759– 769 2, 4,

    2003

  29. [39]

    T2I- Adapter: learning adapters to dig out more controllable ability for text- to-image diffusion models

    [MWX*24] M OU, CHONG , WANG, XINTAO, XIE, LIANGBIN , et al. “T2I- Adapter: learning adapters to dig out more controllable ability for text- to-image diffusion models”.Proceedings of the Thirty-Eighth AAAI Con- ference on Artificial Intelligence and Thirty-Sixth Conference on I...

    2024

  30. [40]

    PyTorch: An Imperative Style, High-Performance Deep Learning Li- brary

    [PGM*19] P ASZKE , A DAM, G ROSS , S AM, M ASSA , F RANCISCO , et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Li- brary”. Advances in Neural Information Processing Systems 32 . Curran Associates, Inc., 2019, 8024–8035 15,

    2019

  31. [41]

    High-resolution image synthesis with latent diffusion models

    [RBL*22] R OMBACH , R OBIN , B LATTMANN , A NDREAS , L ORENZ , D O- MINIK , et al. “High-resolution image synthesis with latent diffusion models”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 10684–10695 3,

    2022

  32. [42]

    2020, 167–178

    Wiley Online Library. 2020, 167–178

    2020

  33. [43]

    2019, 235– 244

    Wiley Online Library. 2019, 235– 244

    2019

  34. [44]

    Learning transferable visual models from natural language super- vision

    [RKH*21] R ADFORD , A LEC, K IM, J ONG WOOK, H ALLACY , C HRIS , et al. “Learning transferable visual models from natural language super- vision”. International Conference on Machine Learning (2021), 8748– 8763 3, 6,

    2021

  35. [45]

    Photorealistic text-to-image diffusion models with deep language understanding

    [SCS*22] S AHARIA , CHITWAN , CHAN, WILLIAM , SAXENA , SAURABH , et al. “Photorealistic text-to-image diffusion models with deep language understanding”. Advances in Neural Information Processing Systems 35 (2022), 36479–36494

    2022

  36. [46]

    The ef- fect of shape and illumination on material perception: model and appli- cations

    [SCW*21] S ERRANO , A NA, C HEN, B IN, WANG, C HAO, et al. “The ef- fect of shape and illumination on material perception: model and appli- cations”. ACM Transactions on Graphics (TOG) 40.4 (2021) 2–5, 15,

    2021

  37. [47]

    An intuitive control space for material appearance

    [SGM*16] S ERRANO , A NA, G UTIERREZ , D IEGO , M YSZKOWSKI , KAROL , et al. “An intuitive control space for material appearance”. ACM Transactions on Graphics (TOG)35.6 (2016)

    2016

  38. [48]

    A mathematical theory of com- munication

    [Sha48] S HANNON , C LAUDE ELWOOD . “A mathematical theory of com- munication”. The Bell System Technical Journal27.3 (1948), 379–423

    1948

  39. [49]

    Alchemist: Parametric control of material properties with diffusion models

    [SJL*24] S HARMA , P RAFULL , J AMPANI , V ARUN , L I, Y UANZHEN , et al. “Alchemist: Parametric control of material properties with diffusion models”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 24130–24141

    2024

  40. [50]

    2023, 333–345 3, 10,

    Wiley Online Library. 2023, 333–345 3, 10,

    2023

  41. [51]

    Ladder variational autoencoders

    [SRM*16] S ØNDERBY , C ASPER KAAE, R AIKO , T APANI , M AALØE , LARS, et al. “Ladder variational autoencoders”. Advances in Neural In- formation Processing Systems 29 (2016) 3,

    2016

  42. [52]

    2018, 135–144 2,

    Wiley Online Library. 2018, 135–144 2,

    2018

  43. [53]

    A low-dimensional perceptual space for intuitive BRDF editing

    [SWSR21] S HI, W EIQI , W ANG, Z EYU, S OLER , C YRIL , and R USH - MEIER , H OLLY. “A low-dimensional perceptual space for intuitive BRDF editing”. EGSR 2021-Eurographics Symposium on Rendering- DL-only Track. 2021, 1–13

    2021

  44. [54]

    Three Perceptual Dimensions for Specu- lar and Diffuse Reflection

    [TGG*20] T OSCANI , M ATTEO , G UARNERA , D AR’YA, G UARNERA , GIUSEPPE CLAUDIO , et al. “Three Perceptual Dimensions for Specu- lar and Diffuse Reflection”. ACM Transactions on Applied Perception 17.2 (May 2020). ISSN : 1544-3558

    2020

  45. [55]

    Collaborative control for geometry-conditioned PBR image genera- tion

    [VBP*24] V AINER , S HIMON , B OSS, M ARK, P ARGER , M ATHIAS , et al. “Collaborative control for geometry-conditioned PBR image genera- tion”. Proceedings of the European Conference on Computer Vision . Springer. 2024, 127–145

    2024

  46. [56]

    ControlMat: A Controlled Generative Approach to Ma- terial Capture

    [VMR*24] V ECCHIO , G IUSEPPE , M ARTIN , R OSALIE , R OULLIER , ARTHUR , et al. “ControlMat: A Controlled Generative Approach to Ma- terial Capture”. ACM Transactions on Graphics (TOG) 43.5 (2024) 2,

    2024

  47. [57]

    Diffusers: State-of-the-art diffusion models

    [vPPL*22] V on P LATEN , PATRICK , PATIL, S URAJ , L OZHKOV , A NTON , et al. Diffusers: State-of-the-art diffusion models. https://github. com/huggingface/diffusers. 2022

    2022

  48. [58]

    Information Theoretical Analysis of Mul- tivariate Correlation

    [Wat60] W ATANABE , SATOSI. “Information Theoretical Analysis of Mul- tivariate Correlation”. IBM Journal of Research and Development 4.1 (1960), 66–82 4,

    1960

  49. [59]

    Image quality assessment: from error visibility to structural similarity

    [WBSS04] W ANG, Z HOU, B OVIK , A.C., S HEIKH , H.R., and S IMON - CELLI , E.P. “Image quality assessment: from error visibility to structural similarity”. IEEE Transactions on Image Processing 13.4 (2004), 600– 612

    2004

  50. [60]

    Disentangled representation learning

    [WCWZ*24] W ANG, X IN, C HEN, H ONG, W U, Z IHAO , Z HU, W ENWU , et al. “Disentangled representation learning”. IEEE Transactions on Pat- tern Analysis and Machine Intelligence (2024)

    2024

  51. [61]

    Disentanglement with Biological Constraints: A Theory of Functional Cell Types

    [WDGB23] W HITTINGTON , JAMES C. R., D ORRELL , WILL, GANGULI , SURYA, and B EHRENS , T IMOTHY . “Disentanglement with Biological Constraints: A Theory of Functional Cell Types”. The Eleventh Interna- tional Conference on Learning Representations, ICLR. 2023

    2023

  52. [62]

    DisDiff: unsupervised disentanglement of diffusion probabilistic models

    [YWLZ23] Y ANG, TAO, WANG, YUWANG , LU, YAN, and ZHENG , NAN- NING . “DisDiff: unsupervised disentanglement of diffusion probabilistic models”. Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023, 69130–69156

    2023

  53. [63]

    Re- balancing variational autoencoder loss for molecule sequence genera- tion

    [YWY*20] Y AN, CHAOCHAO , WANG, SHENG , YANG, JINYU , et al. “Re- balancing variational autoencoder loss for molecule sequence genera- tion”. Proceedings of the 11th ACM international conference on bioin- formatics, computational biology and health informatics. 2020, 1–7

    2020

  54. [64]

    Ip-adapter: Text com- patible image prompt adapter for text-to-image diffusion models

    [YZL*23] Y E, HU, ZHANG , JUN, LIU, SIBO, et al. “Ip-adapter: Text com- patible image prompt adapter for text-to-image diffusion models”. arXiv preprint arXiv:2308.06721 (2023) 2, 3, 8,

    2023 arXiv

  55. [65]

    The unreasonable effectiveness of deep features as a perceptual met- ric

    [ZIE*18] Z HANG , R ICHARD , I SOLA , P HILLIP , E FROS , A LEXEI A, et al. “The unreasonable effectiveness of deep features as a perceptual met- ric”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 586–595

    2018

  56. [66]

    Dreammat: High-quality pbr material generation with geometry-and light-aware dif- fusion models

    14 of 28 S. Jimenez-Navarro, J. Guerrero-Viu & B. Masia / A Controllable Appearance Representation for Flexible Transfer and Editing [ZLX*24] Z HANG , YUQING , LIU, YUAN, XIE, ZHIYU , et al. “Dreammat: High-quality pbr material generation with geometry-and light-aware dif- fus...

    2024

  57. [67]

    Adding conditional control to text-to-image diffusion models

    [ZRA23] Z HANG , L VMIN , R AO, A NYI, and A GRAWALA , M ANEESH . “Adding conditional control to text-to-image diffusion models”. Pro- ceedings of the IEEE/CVF International Conference on Computer Vi- sion. 2023, 3836–3847 3, 8,

    2023

  58. [68]

    A Compact Representation of Measured BRDFs Using Neural Pro- cesses

    [ZZW*21] Z HENG , CHUANKUN , ZHENG , RUZHANG , WANG, RUI, et al. “A Compact Representation of Measured BRDFs Using Neural Pro- cesses”. ACM Transactions on Graphics (TOG) 41.2 (Nov. 2021). ISSN : 0730-0301 2,

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.