Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read MatPredict renders 18 indoor objects in 14 materials and reports that standard neural networks recover their basecolor and roughness maps from a single image.

desk verdict A genuinely new synthetic dataset for object-material appearance, but the benchmark as printed can't support the generalization claim; worth a conditional peer review. read the letter →

arxiv 2505.13201 v1 pith:3QVR65QV submitted 2025-05-19 cs.CV cs.AI

classification cs.CVcs.AI
keywords materialpropertypredictionphysicallybasedrenderinginversesyntheticdatasetindoorroboticsbasecolorestimationroughnessimageregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MatPredict is a synthetic dataset and benchmark for teaching machines to infer visual material properties from a single image. The paper pairs indoor object meshes with a library of physically based materials, renders 18 common objects in 14 materials under varied lighting and viewpoints, and reports that four encoder-decoder models can recover pixel-wise basecolor and roughness maps from a $224\times224$ crop. The motivation is robotic manipulation: knowing whether a mug is ceramic or metal tells a robot how hard to grip it. The paper's contribution is decoupling material diversity from object geometry, so material recognition can be studied without the usual shape-material confound.

What carries the argument

The load-bearing mechanism is the physically based rendering pipeline that makes the image a deterministic function of the material maps. A mesh is paired with a material stack through a Principled BSDF shader (a single node bundling basecolor, roughness, metallic, and related parameters), after a UV pass rescales texel density to object size and packs the UV islands into UDIM tiles to prevent seams. A spherical camera grid and a symmetric ring of area lights produce 512 views per object-material pair. On the regression side, all four networks share an encoder-decoder design whose only material-dependent component is a final $1\times1$ convolution, so basecolor and roughness are predicted jointly and new material channels can be added by changing the output kernel count.

What would settle it

Train on a subset of object identities and test on the remaining identities rendered with the same materials; if per-material error rises sharply for unseen shapes while staying low for seen ones, the reported recovery is shape memorization, not material inference.

Watch

Extended reading notes

Core claim

The central discovery is that a controlled synthetic pipeline can separate material appearance from object shape well enough for current networks to recover the underlying reflectance maps. Each object mesh is paired with one of 14 material stacks from a PBR library and rendered from 512 viewpoints on a fixed spherical grid, after a UV pass equalizes texel density so a teacup and a wardrobe show comparable detail. Benchmarked UNet, ResNet-50, Swin-T, and ConvNeXt-T models regress a six-channel target of basecolor plus roughness; the best combinations reach RMSE around 0.001 and SSIM above 0.99. The paper presents MatPredict as the first dataset that factorises material diversity from object geometry.

Load-bearing premise

The benchmark assumes that holding out viewpoints of the same object-material pairs is enough to prove material learning, so low test error can just mean the network memorized each object's textures rather than learning to infer materials from appearance.

Editorial extensions

If this is right

  • Because geometry and material vary independently in the dataset, robotic perception systems can train material recognition without the confound present in real photographs.
  • The four models, trained with identical splits and schedules, give the community a common baseline for comparing future material-regression methods.
  • Extending the task to metallicity, normal maps, transparency, or other material layers requires only changing the number of output channels in the final $1\times1$ convolution and re-weighting the loss per channel.
  • The deterministic camera grid and lighting rig mean the dataset can be regenerated or extended at different densities without changing the evaluation protocol.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger test the paper does not run is holding out entire object identities or material classes; if error jumps for unseen shapes, low viewpoint-holdout error is partly texture memorization.
  • The reported spectral-angle errors stay near the metric's upper bound even where RMSE is tiny, so per-material and per-channel SAM analysis would clarify whether networks match brightness but miss spectral direction.
  • The current renders assign one uniform material per object; composing several materials on one mesh, as the paper lists as future work, would let the same pipeline support objects like a wooden table with metal legs.
  • Because the ground-truth maps come from the same renderer as the images, the benchmark scores reflect simulator consistency; evaluating on a small set of real photographs would test transfer to physical cameras.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. MatPredict combines 18 object meshes from the Replica dataset with 14 material classes from MatSynth, renders each object-material pair under a controlled lighting rig from 512 viewpoints using Blender Cycles, and releases the renders as a dataset. The paper also presents a benchmark in which ResNet-50 and Swin-T encoder-decoder models regress basecolor and roughness maps from single crops, evaluated with RMSE, SSIM, and SAM (plus a broader metric suite). The central claims are that the pipeline factorises material diversity from object geometry and that the released benchmark demonstrates the feasibility of recovering material properties from images.

Significance. The dataset itself is potentially useful: it is publicly released, the rendering pipeline is described with enough detail (texel-density normalisation, UDIM packing, latitude-longitude camera grid, lighting equations) to be reproduced, and the multi-material-per-object design addresses a real gap in inverse-rendering resources. If the benchmark results were correct and the split design supported generalisation claims, the paper would make a solid contribution to indoor robotics perception. However, the quantitative evidence as printed contains internal inconsistencies, and the evaluation protocol does not support the paper's strongest claims; these issues must be resolved before the benchmark can be trusted.

major comments (4)
  1. [Tables 4 and 6; §4.5] Table 4 and Table 6 report SAM values around 87–90 degrees alongside RMSE below 0.007 and SSIM above 0.94 in almost every row (e.g., Table 4, Wood, ResNet-50: RMSE 0.0025, SSIM 0.9883, SAM 87.70). This is internally inconsistent: for two images that agree to within 0.0025 RMSE and 0.99 SSIM, the spectral angle mapper should be near zero, not near the top of its range; an angle of ~88 degrees means the colour vectors are nearly orthogonal. The SAM values therefore appear to be computed incorrectly or reported against the wrong reference. Since Table 4 is the main quantitative support for the paper's benchmark claim, the metric definitions and the reported numbers need to be checked and recomputed.
  2. [§4.3] Section 4.3 splits the 512 screenshots of each object-material pair 80/20 by random seed, so every test image shares both its object identity and its material with training images. Because the regression targets are exactly the basecolor and roughness maps used to render those pairs, the task can be solved by recognising the pair from coarse shape or context and emitting the memorised texture map; with only 18×14=252 pairs and all pairs present in training, the reported low error does not demonstrate material inference from appearance. The claim in Section 5 that MatPredict 'factorises material diversity from object geometry' is therefore not tested. Please add leave-one-object-out or leave-one-material-out splits, or a real-image transfer evaluation, and report generalization separately from the in-distribution viewpoint split.
  3. [§4.1–4.2 and Tables 4, 6] Section 4.1 and Table 1 advertise a four-model benchmark (UNet-no-skip, ResNet-50, Swin-T, ConvNeXt-T), but Tables 4 and 6 report results only for ResNet-50 and Swin-T. The paper does not explain why the other two models are missing. As written, the claim of a 'four-model benchmark' is unsupported by the experimental section.
  4. [§5] Section 5 states that MATPREDICT is 'the first dataset that factorises material diversity from object geometry'. The dataset construction does pair each object with multiple materials, but the benchmark never varies geometry and material independently at test time; no experiment holds geometry fixed while changing material, or vice versa. The novelty claim should be reworded to describe the dataset's construction, not the demonstrated learning result, until such experiments are provided.
minor comments (6)
  1. [Table 6] Table 6, Concrete row: '0.0.0021' contains a typo and should presumably read '0.0021'.
  2. [§4.3] The sentence 'we shuffle S once and split it 80 SUBSET' is incomplete; specify the exact train/validation/test proportions and how the validation set is used.
  3. [Appendix A, Table 5] Table 5 gives the SAM range as [0, π/2] rad, while Tables 4 and 6 report SAM in degrees; please be consistent about angular units.
  4. [§3] The text reads 'MatSynth contains from than 4000' and should read 'contains more than 4000'.
  5. [Table 3] Table 3 is labelled 'partial' but no full 18×14 matrix is supplied; either provide the complete matrix or state clearly how the six shown rows were selected.
  6. [§4.3] The code fragment 'we keep–if_cropped False' should be replaced by a prose description of the cropping behaviour.

Circularity Check

2 steps flagged · score 6.0 of 10

Benchmark 'predictions' are the exact texture maps fed into the renderer, and the held-out-viewpoint split keeps every object-material pair's target map in training, so the low error does not demonstrate general material inference.

  1. fitted input called prediction [Section 3 (dataset generation) and Section 4.3, Eq. (7)]
    "we query the MatSynth dataset and retrieve the calibrated texture bundle associated with every material class—namely basecolor, diffuse, metallic, normal, opacity, and roughness maps. Each sub-mesh is then paired with the texture set ... into a Principled BSDF shader during Cycles rendering. ... For each sample we stack the basecolour and roughness maps channel-wise to obtain a 6-channel target T= [B∥R] ... The training objective is ... L=∥Bˆ−B∥^2_2 + ∥Rˆ−R∥^2_2."

    The basecolour B and roughness R maps used as prediction targets in Eq. (7) are exactly the texture maps inserted into the Principled BSDF shader to render the input image. The benchmark therefore measures whether a network can invert the paper's own forward rendering model on its own training distribution, not whether it discovers or predicts material properties as an independent quantity. Low RMSE/SSIM is expected by construction if the renderer is invertible, so the reported numbers cannot support the broader claim that MATPREDICT factorises material diversity from object geometry.

  2. fitted input called prediction [Section 4.3, Dataset preparation and train/test split]
    "For every material class ⟨m⟩∈M and object category⟨o⟩∈O the simulator exports •512RGB screenshotsrendered_cropped/⟨o⟩/⟨m⟩/*.png, • one referencebasecolourmapground_truth_basecolour/⟨m⟩.png, • one referenceroughnessmapground_truth_roughness/⟨m⟩.png. ... With a fixed seed (42) we shuffle S once and split it 80 SUBSET."

    Because the ground-truth basecolour and roughness maps are stored once per material class, all 512 screenshots of an object-material pair share the identical target map. Splitting the 512 screenshot indices 80/20 means every test image belongs to an object-material pair whose target map also appears in the training set. A network can memorize the 18×14 pair-to-map associations and retrieve the correct map at test time without performing viewpoint-invariant material inference. The reported near-perfect errors are therefore statistically forced by pair identity, and the benchmark does not test the claimed factorization on unseen objects, materials, or object-material combinations.

full rationale

The dataset construction itself is not circular: MATPREDICT genuinely combines Replica meshes with MatSynth materials, and the paper does not rely on self-citations or imported uniqueness theorems. The circularity lies in the benchmark's predictive claim. The targets B and R in Eq. (7) are the very maps fed into the Principled BSDF shader that produced the input images, so the 'prediction' is the inverse of the paper's own known forward model. Compounding this, the data preparation stores one target map per material class, and the train/test split is over screenshot indices rather than over objects or materials. Every test image therefore shares its object-material pair and its exact target map with training, enabling trivial pair-to-map memorization. The central novelty claim that MATPREDICT 'factorises material diversity from object geometry' is never evaluated because no experiment transfers to an unseen object, unseen material, or unseen object-material pair. Additionally, the reported SAM values of 87–90 degrees alongside RMSE ~0.002 and SSIM ~0.99 are internally inconsistent for near-identical images, further undermining the printed benchmark evidence. For these reasons, the central benchmark result reduces by construction to renderer inversion plus per-pair memorization, warranting a circularity score of 6 rather than a lower score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No parameters are fitted to data in the benchmark; the listed free parameters are hand-chosen design settings for the dataset generation. The axioms are domain assumptions about rendering fidelity, material representativeness, and problem well-posedness that the central claim rests on but does not independently verify.

free parameters (2)
  • texel density d0 = 512 px/m at reference length 1 m
    Hand-selected in UV preprocessing to normalize texel density across objects; affects texture scale but is not fitted to data and does not determine benchmark outcome.
  • lamp power coefficients = P_base = 50 + 150 * min(1, r/1 m)
    Hand-tuned formula for lamp power as a function of object radius; a dataset design choice, not fitted to any target.
assumptions (3)
  • domain assumption Blender Cycles accurately simulates light-material interaction according to the physically based rendering equation.
    The dataset and benchmark rely on the fidelity of the renderer to produce images from which material properties can be recovered; this assumption is stated implicitly in Section 3.
  • domain assumption MatSynth material maps (basecolor, roughness, normal, etc.) are representative of real-world material classes.
    The benchmark treats these maps as ground truth for material properties, which is reasonable for synthetic data but assumes the maps capture what the network should learn; cited in Section 3.
  • domain assumption The rendered image combined with the basecolor and roughness targets forms a well-posed inverse rendering task that networks can solve.
    The benchmark implicitly assumes that the inverse problem is learnable from the provided viewpoints and lighting, which is not proven and is challenged by the limited generalization evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects." pith.science (2026). https://pith.science/paper/3QVR65QV

@misc{pith2026250513201,
  author       = {Pith},
  title        = {Pith review of: MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3QVR65QV}},
  note         = {Machine review of arXiv:2505.13201}
}
read the original abstract

Determining material properties from camera images can expand the ability to identify complex objects in indoor environments, which is valuable for consumer robotics applications. To support this, we introduce MatPredict, a dataset that combines the high-quality synthetic objects from Replica dataset with MatSynth dataset's material properties classes - to create objects with diverse material properties. We select 3D meshes of specific foreground objects and render them with different material properties. In total, we generate \textbf{18} commonly occurring objects with \textbf{14} different materials. We showcase how we provide variability in terms of lighting and camera placement for these objects. Next, we provide a benchmark for inferring material properties from visual images using these perturbed models in the scene, discussing the specific neural network models involved and their performance based on different image comparison metrics. By accurately simulating light interactions with different materials, we can enhance realism, which is crucial for training models effectively through large-scale simulations. This research aims to revolutionize perception in consumer robotics. The dataset is provided \href{https://huggingface.co/datasets/UMTRI/MatPredict}{here} and the code is provided \href{https://github.com/arpan-kusari/MatPredict}{here}.

Figures

Figures reproduced from arXiv: 2505.13201 by the authors.

Figure 1
Figure 1. Example glass decorative pieces Material properties through visual identification form a reliable way of interacting with unknown objects in the real world. For example, identification of fragile items helps determine the force and the touch points when handled by robots (different examples of glass items shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Table rendered by different materials We curate a large scale dataset aimed towards creating different versions of the same object based on different material properties. We utilize Replica, introduced by Meta, which is a dataset of 18 highly photo-realistic 3D indoor scene reconstructions [20]. Each scene in the dataset consists of a dense mesh, high-resolution high-dynamic-range (HDR) textures, per-primitive seman… view at source ↗
Figure 3
Figure 3. Effect of the UV scale factor s on appearance. Smaller s values map a larger texture footprint to the surface, producing finer weave patterns, whereas larger s values stretch the same fabric map, leading to visibly coarser detail. UV texture preprocessing. Before any rendered frame enters our pipeline we perform a single, automated UV-editing pass that prepares every sub-mesh for texture look-up. First, a texel-dens… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Effect of viewpoint on appearance. Five leather-pillow renderings sampled from the [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: leather pillow with different lighting setup [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WCM: World-Cognition Model for Generalizable Human-Robot Interaction

    cs.RO 2026-07 conditional novelty 6.0 of 10

    WCM, a modular robot system with asynchronous execution and chain-of-thought distillation, reports 73.8% success across nine interactive tasks and learns a long-horizon task from natural-language teaching.

Reference graph

Works this paper leans on

26 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    A K Peters / CRC Press, Boca Raton, FL, 2018

    Tomas Akenine-Möller, Eric Haines, Naty Hoffman, Angelo Pesce, Michał Iwanicki, and Sébastien Hillaire.Real-Time Rendering (4th Edition). A K Peters / CRC Press, Boca Raton, FL, 2018

  2. [2]

    A similarity measure for material appearance.Jornada de Jóvenes Investigadores del I3A, 7, 2019

    Manuel Lagunas Arto, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. A similarity measure for material appearance.Jornada de Jóvenes Investigadores del I3A, 7, 2019

  3. [3]

    Blender Foundation, 2024

    Blender Documentation Team.Mapping Types — Smart UV Project.Blender Manual. Blender Foundation, 2024. Accessed May 16, 2025

  4. [4]

    Invariant scattering convolution networks.IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013

    Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks.IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013

  5. [5]

    Pcanet: A simple deep learning baseline for image classification?IEEE transactions on image processing, 24(12):5017–5032, 2015

    Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma. Pcanet: A simple deep learning baseline for image classification?IEEE transactions on image processing, 24(12):5017–5032, 2015

  6. [6]

    Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018

    Blender Online Community.Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018

  7. [7]

    Visual perception of materials and their properties.Vision research, 94:62–75, 2014

    Roland W Fleming. Visual perception of materials and their properties.Vision research, 94:62–75, 2014

  8. [8]

    Perceptual qualities and material classes.Journal of vision, 13(8):9–9, 2013

    Roland W Fleming, Christiane Wiebel, and Karl Gegenfurtner. Perceptual qualities and material classes.Journal of vision, 13(8):9–9, 2013

Show all 26 references
  1. [9]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016

  2. [10]

    Mitsuba 3 renderer,

    Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. Mitsuba 3 renderer,

  3. [11]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023

  4. [12]

    Representing and recognizing the visual appearance of materials using three-dimensional textons.International journal of computer vision, 43(1):29– 44, 2001

    Thomas Leung and Jitendra Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons.International journal of computer vision, 43(1):29– 44, 2001

  5. [13]

    Cgintrinsics: Better intrinsic image decomposition through physically-based rendering

    Zhengqi Li and Noah Snavely. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. InProceedings of the European conference on computer vision (ECCV), pages 371–387, 2018

  6. [14]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, 2021

  7. [15]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976–11986, 2022

  8. [16]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical Image Computing and Computer-Assisted Interven- tion (MICCAI), volume 9351 ofLecture Notes in Computer Science, pages 234–241. Springer, 2015

  9. [17]

    Recognizing material properties from images.IEEE transactions on pattern analysis and machine intelligence, 42(8):1981–1995, 2019

    Gabriel Schwartz and Ko Nishino. Recognizing material properties from images.IEEE transactions on pattern analysis and machine intelligence, 42(8):1981–1995, 2019. 10

  10. [18]

    Neural inverse rendering of an indoor scene from a single image

    Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8598–8607, 2019

  11. [19]

    Semantic scene completion from a single depth image

    Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1746–1754, 2017

  12. [20]

    The replica dataset: A digital replica of indoor spaces.arXiv preprint arXiv:1906.05797, 2019

    Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces.arXiv preprint arXiv:1906.05797, 2019

  13. [21]

    Deep learning for material recognition: most recent advances and open challenges.arXiv preprint arXiv:2012.07495, 2020

    Alain Trémeau, Sixiang Xu, and Damien Muselet. Deep learning for material recognition: most recent advances and open challenges.arXiv preprint arXiv:2012.07495, 2020

  14. [22]

    Texture analysis.Handbook of pattern recognition and computer vision, pages 235–276, 1993

    Mihran Tuceryan and Anil K Jain. Texture analysis.Handbook of pattern recognition and computer vision, pages 235–276, 1993

  15. [23]

    Matsynth: A modern pbr materials dataset.arXiv preprint arXiv:2401.06056, 2024

    Giuseppe Vecchio and Valentin Deschaintre. Matsynth: A modern pbr materials dataset.arXiv preprint arXiv:2401.06056, 2024

  16. [24]

    Pyramidal parametrics

    Lance Williams. Pyramidal parametrics. InProceedings of the 10th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’83), pages 1–11. ACM, 1983

  17. [25]

    Physically-based rendering for indoor scene understanding using convolu- tional neural networks

    Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon-Young Lee, Hailin Jin, and Thomas Funkhouser. Physically-based rendering for indoor scene understanding using convolu- tional neural networks. InProceedings of the IEEE conference on computer vision and pattern recogni...

  18. [2022]

    https://mitsuba-renderer.org

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.