REVIEW 4 major objections 6 minor 1 cited by
MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read MatPredict renders 18 indoor objects in 14 materials and reports that standard neural networks recover their basecolor and roughness maps from a single image.
desk verdict A genuinely new synthetic dataset for object-material appearance, but the benchmark as printed can't support the generalization claim; worth a conditional peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the physically based rendering pipeline that makes the image a deterministic function of the material maps. A mesh is paired with a material stack through a Principled BSDF shader (a single node bundling basecolor, roughness, metallic, and related parameters), after a UV pass rescales texel density to object size and packs the UV islands into UDIM tiles to prevent seams. A spherical camera grid and a symmetric ring of area lights produce 512 views per object-material pair. On the regression side, all four networks share an encoder-decoder design whose only material-dependent component is a final $1\times1$ convolution, so basecolor and roughness are predicted jointly and new material channels can be added by changing the output kernel count.
What would settle it
Train on a subset of object identities and test on the remaining identities rendered with the same materials; if per-material error rises sharply for unseen shapes while staying low for seen ones, the reported recovery is shape memorization, not material inference.
Extended reading notes
Core claim
The central discovery is that a controlled synthetic pipeline can separate material appearance from object shape well enough for current networks to recover the underlying reflectance maps. Each object mesh is paired with one of 14 material stacks from a PBR library and rendered from 512 viewpoints on a fixed spherical grid, after a UV pass equalizes texel density so a teacup and a wardrobe show comparable detail. Benchmarked UNet, ResNet-50, Swin-T, and ConvNeXt-T models regress a six-channel target of basecolor plus roughness; the best combinations reach RMSE around 0.001 and SSIM above 0.99. The paper presents MatPredict as the first dataset that factorises material diversity from object geometry.
Load-bearing premise
The benchmark assumes that holding out viewpoints of the same object-material pairs is enough to prove material learning, so low test error can just mean the network memorized each object's textures rather than learning to infer materials from appearance.
Editorial extensions
If this is right
- Because geometry and material vary independently in the dataset, robotic perception systems can train material recognition without the confound present in real photographs.
- The four models, trained with identical splits and schedules, give the community a common baseline for comparing future material-regression methods.
- Extending the task to metallicity, normal maps, transparency, or other material layers requires only changing the number of output channels in the final $1\times1$ convolution and re-weighting the loss per channel.
- The deterministic camera grid and lighting rig mean the dataset can be regenerated or extended at different densities without changing the evaluation protocol.
Reading between the lines
- A stronger test the paper does not run is holding out entire object identities or material classes; if error jumps for unseen shapes, low viewpoint-holdout error is partly texture memorization.
- The reported spectral-angle errors stay near the metric's upper bound even where RMSE is tiny, so per-material and per-channel SAM analysis would clarify whether networks match brightness but miss spectral direction.
- The current renders assign one uniform material per object; composing several materials on one mesh, as the paper lists as future work, would let the same pipeline support objects like a wooden table with metal legs.
- Because the ground-truth maps come from the same renderer as the images, the benchmark scores reflect simulator consistency; evaluating on a small set of real photographs would test transfer to physical cameras.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. MatPredict combines 18 object meshes from the Replica dataset with 14 material classes from MatSynth, renders each object-material pair under a controlled lighting rig from 512 viewpoints using Blender Cycles, and releases the renders as a dataset. The paper also presents a benchmark in which ResNet-50 and Swin-T encoder-decoder models regress basecolor and roughness maps from single crops, evaluated with RMSE, SSIM, and SAM (plus a broader metric suite). The central claims are that the pipeline factorises material diversity from object geometry and that the released benchmark demonstrates the feasibility of recovering material properties from images.
Significance. The dataset itself is potentially useful: it is publicly released, the rendering pipeline is described with enough detail (texel-density normalisation, UDIM packing, latitude-longitude camera grid, lighting equations) to be reproduced, and the multi-material-per-object design addresses a real gap in inverse-rendering resources. If the benchmark results were correct and the split design supported generalisation claims, the paper would make a solid contribution to indoor robotics perception. However, the quantitative evidence as printed contains internal inconsistencies, and the evaluation protocol does not support the paper's strongest claims; these issues must be resolved before the benchmark can be trusted.
major comments (4)
- [Tables 4 and 6; §4.5] Table 4 and Table 6 report SAM values around 87–90 degrees alongside RMSE below 0.007 and SSIM above 0.94 in almost every row (e.g., Table 4, Wood, ResNet-50: RMSE 0.0025, SSIM 0.9883, SAM 87.70). This is internally inconsistent: for two images that agree to within 0.0025 RMSE and 0.99 SSIM, the spectral angle mapper should be near zero, not near the top of its range; an angle of ~88 degrees means the colour vectors are nearly orthogonal. The SAM values therefore appear to be computed incorrectly or reported against the wrong reference. Since Table 4 is the main quantitative support for the paper's benchmark claim, the metric definitions and the reported numbers need to be checked and recomputed.
- [§4.3] Section 4.3 splits the 512 screenshots of each object-material pair 80/20 by random seed, so every test image shares both its object identity and its material with training images. Because the regression targets are exactly the basecolor and roughness maps used to render those pairs, the task can be solved by recognising the pair from coarse shape or context and emitting the memorised texture map; with only 18×14=252 pairs and all pairs present in training, the reported low error does not demonstrate material inference from appearance. The claim in Section 5 that MatPredict 'factorises material diversity from object geometry' is therefore not tested. Please add leave-one-object-out or leave-one-material-out splits, or a real-image transfer evaluation, and report generalization separately from the in-distribution viewpoint split.
- [§4.1–4.2 and Tables 4, 6] Section 4.1 and Table 1 advertise a four-model benchmark (UNet-no-skip, ResNet-50, Swin-T, ConvNeXt-T), but Tables 4 and 6 report results only for ResNet-50 and Swin-T. The paper does not explain why the other two models are missing. As written, the claim of a 'four-model benchmark' is unsupported by the experimental section.
- [§5] Section 5 states that MATPREDICT is 'the first dataset that factorises material diversity from object geometry'. The dataset construction does pair each object with multiple materials, but the benchmark never varies geometry and material independently at test time; no experiment holds geometry fixed while changing material, or vice versa. The novelty claim should be reworded to describe the dataset's construction, not the demonstrated learning result, until such experiments are provided.
minor comments (6)
- [Table 6] Table 6, Concrete row: '0.0.0021' contains a typo and should presumably read '0.0021'.
- [§4.3] The sentence 'we shuffle S once and split it 80 SUBSET' is incomplete; specify the exact train/validation/test proportions and how the validation set is used.
- [Appendix A, Table 5] Table 5 gives the SAM range as [0, π/2] rad, while Tables 4 and 6 report SAM in degrees; please be consistent about angular units.
- [§3] The text reads 'MatSynth contains from than 4000' and should read 'contains more than 4000'.
- [Table 3] Table 3 is labelled 'partial' but no full 18×14 matrix is supplied; either provide the complete matrix or state clearly how the six shown rows were selected.
- [§4.3] The code fragment 'we keep–if_cropped False' should be replaced by a prose description of the cropping behaviour.
Circularity Check
Benchmark 'predictions' are the exact texture maps fed into the renderer, and the held-out-viewpoint split keeps every object-material pair's target map in training, so the low error does not demonstrate general material inference.
-
fitted input called prediction
[Section 3 (dataset generation) and Section 4.3, Eq. (7)]
"we query the MatSynth dataset and retrieve the calibrated texture bundle associated with every material class—namely basecolor, diffuse, metallic, normal, opacity, and roughness maps. Each sub-mesh is then paired with the texture set ... into a Principled BSDF shader during Cycles rendering. ... For each sample we stack the basecolour and roughness maps channel-wise to obtain a 6-channel target T= [B∥R] ... The training objective is ... L=∥Bˆ−B∥^2_2 + ∥Rˆ−R∥^2_2."
The basecolour B and roughness R maps used as prediction targets in Eq. (7) are exactly the texture maps inserted into the Principled BSDF shader to render the input image. The benchmark therefore measures whether a network can invert the paper's own forward rendering model on its own training distribution, not whether it discovers or predicts material properties as an independent quantity. Low RMSE/SSIM is expected by construction if the renderer is invertible, so the reported numbers cannot support the broader claim that MATPREDICT factorises material diversity from object geometry.
-
fitted input called prediction
[Section 4.3, Dataset preparation and train/test split]
"For every material class ⟨m⟩∈M and object category⟨o⟩∈O the simulator exports •512RGB screenshotsrendered_cropped/⟨o⟩/⟨m⟩/*.png, • one referencebasecolourmapground_truth_basecolour/⟨m⟩.png, • one referenceroughnessmapground_truth_roughness/⟨m⟩.png. ... With a fixed seed (42) we shuffle S once and split it 80 SUBSET."
Because the ground-truth basecolour and roughness maps are stored once per material class, all 512 screenshots of an object-material pair share the identical target map. Splitting the 512 screenshot indices 80/20 means every test image belongs to an object-material pair whose target map also appears in the training set. A network can memorize the 18×14 pair-to-map associations and retrieve the correct map at test time without performing viewpoint-invariant material inference. The reported near-perfect errors are therefore statistically forced by pair identity, and the benchmark does not test the claimed factorization on unseen objects, materials, or object-material combinations.
full rationale
The dataset construction itself is not circular: MATPREDICT genuinely combines Replica meshes with MatSynth materials, and the paper does not rely on self-citations or imported uniqueness theorems. The circularity lies in the benchmark's predictive claim. The targets B and R in Eq. (7) are the very maps fed into the Principled BSDF shader that produced the input images, so the 'prediction' is the inverse of the paper's own known forward model. Compounding this, the data preparation stores one target map per material class, and the train/test split is over screenshot indices rather than over objects or materials. Every test image therefore shares its object-material pair and its exact target map with training, enabling trivial pair-to-map memorization. The central novelty claim that MATPREDICT 'factorises material diversity from object geometry' is never evaluated because no experiment transfers to an unseen object, unseen material, or unseen object-material pair. Additionally, the reported SAM values of 87–90 degrees alongside RMSE ~0.002 and SSIM ~0.99 are internally inconsistent for near-identical images, further undermining the printed benchmark evidence. For these reasons, the central benchmark result reduces by construction to renderer inversion plus per-pair memorization, warranting a circularity score of 6 rather than a lower score.
Assumptions & free parameters
free parameters (2)
- texel density d0 =
512 px/m at reference length 1 m
- lamp power coefficients =
P_base = 50 + 150 * min(1, r/1 m)
assumptions (3)
- domain assumption Blender Cycles accurately simulates light-material interaction according to the physically based rendering equation.
- domain assumption MatSynth material maps (basecolor, roughness, normal, etc.) are representative of real-world material classes.
- domain assumption The rendered image combined with the basecolor and roughness targets forms a well-posed inverse rendering task that networks can solve.
Cite this review
Pith. "Pith review of MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects." pith.science (2026). https://pith.science/paper/3QVR65QV
@misc{pith2026250513201,
author = {Pith},
title = {Pith review of: MatPredict: a dataset and benchmark for learning material properties of diverse indoor objects},
year = {2026},
howpublished = {\url{https://pith.science/paper/3QVR65QV}},
note = {Machine review of arXiv:2505.13201}
}
read the original abstract
Determining material properties from camera images can expand the ability to identify complex objects in indoor environments, which is valuable for consumer robotics applications. To support this, we introduce MatPredict, a dataset that combines the high-quality synthetic objects from Replica dataset with MatSynth dataset's material properties classes - to create objects with diverse material properties. We select 3D meshes of specific foreground objects and render them with different material properties. In total, we generate \textbf{18} commonly occurring objects with \textbf{14} different materials. We showcase how we provide variability in terms of lighting and camera placement for these objects. Next, we provide a benchmark for inferring material properties from visual images using these perturbed models in the scene, discussing the specific neural network models involved and their performance based on different image comparison metrics. By accurately simulating light interactions with different materials, we can enhance realism, which is crucial for training models effectively through large-scale simulations. This research aims to revolutionize perception in consumer robotics. The dataset is provided \href{https://huggingface.co/datasets/UMTRI/MatPredict}{here} and the code is provided \href{https://github.com/arpan-kusari/MatPredict}{here}.
Figures
Forward citations
Cited by 1 Pith paper
-
WCM: World-Cognition Model for Generalizable Human-Robot Interaction
WCM, a modular robot system with asynchronous execution and chain-of-thought distillation, reports 73.8% success across nine interactive tasks and learns a long-horizon task from natural-language teaching.
Reference graph
Works this paper leans on
-
[1]
A K Peters / CRC Press, Boca Raton, FL, 2018
Tomas Akenine-Möller, Eric Haines, Naty Hoffman, Angelo Pesce, Michał Iwanicki, and Sébastien Hillaire.Real-Time Rendering (4th Edition). A K Peters / CRC Press, Boca Raton, FL, 2018
work page 2018
-
[2]
A similarity measure for material appearance.Jornada de Jóvenes Investigadores del I3A, 7, 2019
Manuel Lagunas Arto, Sandra Malpica, Ana Serrano, Elena Garces, Diego Gutierrez, and Belen Masia. A similarity measure for material appearance.Jornada de Jóvenes Investigadores del I3A, 7, 2019
work page 2019
-
[3]
Blender Documentation Team.Mapping Types — Smart UV Project.Blender Manual. Blender Foundation, 2024. Accessed May 16, 2025
work page 2024
-
[4]
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks.IEEE transactions on pattern analysis and machine intelligence, 35(8):1872–1886, 2013
work page 2013
-
[5]
Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma. Pcanet: A simple deep learning baseline for image classification?IEEE transactions on image processing, 24(12):5017–5032, 2015
work page 2015
-
[6]
Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
Blender Online Community.Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
2018
-
[7]
Visual perception of materials and their properties.Vision research, 94:62–75, 2014
Roland W Fleming. Visual perception of materials and their properties.Vision research, 94:62–75, 2014
work page 2014
-
[8]
Perceptual qualities and material classes.Journal of vision, 13(8):9–9, 2013
Roland W Fleming, Christiane Wiebel, and Karl Gegenfurtner. Perceptual qualities and material classes.Journal of vision, 13(8):9–9, 2013
work page 2013
Show all 26 references
-
[9]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InIEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016
2016
-
[10]
Mitsuba 3 renderer,
Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. Mitsuba 3 renderer,
-
[11]
Segment anything
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023
2023
-
[12]
Representing and recognizing the visual appearance of materials using three-dimensional textons.International journal of computer vision, 43(1):29– 44, 2001
Thomas Leung and Jitendra Malik. Representing and recognizing the visual appearance of materials using three-dimensional textons.International journal of computer vision, 43(1):29– 44, 2001
2001
-
[13]
Cgintrinsics: Better intrinsic image decomposition through physically-based rendering
Zhengqi Li and Noah Snavely. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. InProceedings of the European conference on computer vision (ECCV), pages 371–387, 2018
2018
-
[14]
Swin transformer: Hierarchical vision transformer using shifted windows
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. InIEEE/CVF International Conference on Computer Vision (ICCV), pages 10012–10022, 2021
2021
-
[15]
A convnet for the 2020s
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11976–11986, 2022
2022
-
[16]
U-net: Convolutional networks for biomedical image segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. InMedical Image Computing and Computer-Assisted Interven- tion (MICCAI), volume 9351 ofLecture Notes in Computer Science, pages 234–241. Springer, 2015
2015
-
[17]
Recognizing material properties from images.IEEE transactions on pattern analysis and machine intelligence, 42(8):1981–1995, 2019
Gabriel Schwartz and Ko Nishino. Recognizing material properties from images.IEEE transactions on pattern analysis and machine intelligence, 42(8):1981–1995, 2019. 10
1981
-
[18]
Neural inverse rendering of an indoor scene from a single image
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 8598–8607, 2019
2019
-
[19]
Semantic scene completion from a single depth image
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1746–1754, 2017
2017
-
[20]
The replica dataset: A digital replica of indoor spaces.arXiv preprint arXiv:1906.05797, 2019
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces.arXiv preprint arXiv:1906.05797, 2019
1906 arXiv
-
[21]
Deep learning for material recognition: most recent advances and open challenges.arXiv preprint arXiv:2012.07495, 2020
Alain Trémeau, Sixiang Xu, and Damien Muselet. Deep learning for material recognition: most recent advances and open challenges.arXiv preprint arXiv:2012.07495, 2020
2012 arXiv
-
[22]
Texture analysis.Handbook of pattern recognition and computer vision, pages 235–276, 1993
Mihran Tuceryan and Anil K Jain. Texture analysis.Handbook of pattern recognition and computer vision, pages 235–276, 1993
1993
-
[23]
Matsynth: A modern pbr materials dataset.arXiv preprint arXiv:2401.06056, 2024
Giuseppe Vecchio and Valentin Deschaintre. Matsynth: A modern pbr materials dataset.arXiv preprint arXiv:2401.06056, 2024
2024 arXiv
-
[24]
Pyramidal parametrics
Lance Williams. Pyramidal parametrics. InProceedings of the 10th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’83), pages 1–11. ACM, 1983
1983
-
[25]
Physically-based rendering for indoor scene understanding using convolu- tional neural networks
Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon-Young Lee, Hailin Jin, and Thomas Funkhouser. Physically-based rendering for indoor scene understanding using convolu- tional neural networks. InProceedings of the IEEE conference on computer vision and pattern recogni...
2017
-
[2022]
https://mitsuba-renderer.org
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.