{"id":"ab1a8000-0077-4ec5-88a9-94abaffc3838","arxiv_id":"2505.13201","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A dataset of rendered indoor objects with varied materials, plus a benchmark for predicting material color and roughness maps from images, with benchmark numbers that contradict their own error metrics.","lead":"The paper creates a small synthetic dataset that renders 18 indoor object shapes with 14 material appearances and tests how well four neural networks recover the color and roughness maps that define each material. The dataset could be a useful resource for robotics, but the benchmark numbers are internally inconsistent and the generalization claims are untested.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Held-out-viewpoint split cannot support the central generalization claim; with only 18 objects and 14 materials, a model can memorize per-pair texture maps, and no cross-object, cross-material, or real-image evaluation is provided.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the viewpoint-only split does not separate material inference from texture memorization. I agree with that diagnosis. The paper's stated contribution is a dataset that factorises material diversity from object geometry, yet the benchmark never evaluates transfer across geometry; all object-material pairs are present in training, so low test error can be explained by memorizing the pair-specific target maps. This is a missing experiment rather than a claimed result that is internally contradictory, so it supports a conditional verdict pending a cross-object or cross-material evaluation. The additional anomaly in the SAM column (reported near 90 degrees despite near-zero RMSE) is a concrete sign that the quantitative evidence needs correction, but it is secondary to the split problem because even corrected metrics would not establish generalization. Thus the reader's CONDITIONAL verdict should be maintained unchanged.","tokens_in":8920,"tokens_out":9039,"duration_ms":93233,"concrete_test":"Re-run the benchmark with a leave-one-object-out split using the released code: train on all 512 views of every object except one and evaluate on all views of the held-out object (same materials). Compare held-out-object RMSE/SSIM with the within-pair Table 4 numbers; if performance degrades substantially, the reported error reflects memorization of per-pair textures rather than material inference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 splits the 512 screenshots of each object-material pair 80/20 by random seed, so every test image shares its object and material with training images. Because the ground-truth basecolor and roughness maps are exactly the maps used to render each pair, the target for a given pair is effectively the same texture map across all viewpoints. A model can identify the pair from coarse shape/context and emit a memorized texture map, achieving the low RMSE and high SSIM in Table 4 without performing material inference from appearance. There are at most 18x14 = 252 pairs, and the training set contains all of them. The central novelty claim in Section 5, that MATPREDICT 'factorises material diversity from object geometry', is therefore not tested: no experiment measures transfer to an unseen object, unseen material, or unseen object-material combination, and the acknowledged synthetic-to-real domain gap is never evaluated. The contradictory SAM values in Table 4 (87-90 degrees alongside RMSE ~0.002 and SSIM ~0.99, where near-identical images should give SAM near 0) reinforce that the printed benchmark evidence is unreliable as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MatPredict combines 18 object meshes from the Replica dataset with 14 material classes from MatSynth, renders each object-material pair under a controlled lighting rig from 512 viewpoints using Blender Cycles, and releases the renders as a dataset. The paper also presents a benchmark in which ResNet-50 and Swin-T encoder-decoder models regress basecolor and roughness maps from single crops, evaluated with RMSE, SSIM, and SAM (plus a broader metric suite). The central claims are that the pipeline factorises material diversity from object geometry and that the released benchmark demonstrates the feasibility of recovering material properties from images.","tokens_in":9151,"tokens_out":5438,"duration_ms":49518,"significance":"The dataset itself is potentially useful: it is publicly released, the rendering pipeline is described with enough detail (texel-density normalisation, UDIM packing, latitude-longitude camera grid, lighting equations) to be reproduced, and the multi-material-per-object design addresses a real gap in inverse-rendering resources. If the benchmark results were correct and the split design supported generalisation claims, the paper would make a solid contribution to indoor robotics perception. However, the quantitative evidence as printed contains internal inconsistencies, and the evaluation protocol does not support the paper's strongest claims; these issues must be resolved before the benchmark can be trusted.","major_comments":[{"comment":"Table 4 and Table 6 report SAM values around 87–90 degrees alongside RMSE below 0.007 and SSIM above 0.94 in almost every row (e.g., Table 4, Wood, ResNet-50: RMSE 0.0025, SSIM 0.9883, SAM 87.70). This is internally inconsistent: for two images that agree to within 0.0025 RMSE and 0.99 SSIM, the spectral angle mapper should be near zero, not near the top of its range; an angle of ~88 degrees means the colour vectors are nearly orthogonal. The SAM values therefore appear to be computed incorrectly or reported against the wrong reference. Since Table 4 is the main quantitative support for the paper's benchmark claim, the metric definitions and the reported numbers need to be checked and recomputed.","section":"Tables 4 and 6; §4.5"},{"comment":"Section 4.3 splits the 512 screenshots of each object-material pair 80/20 by random seed, so every test image shares both its object identity and its material with training images. Because the regression targets are exactly the basecolor and roughness maps used to render those pairs, the task can be solved by recognising the pair from coarse shape or context and emitting the memorised texture map; with only 18×14=252 pairs and all pairs present in training, the reported low error does not demonstrate material inference from appearance. The claim in Section 5 that MatPredict 'factorises material diversity from object geometry' is therefore not tested. Please add leave-one-object-out or leave-one-material-out splits, or a real-image transfer evaluation, and report generalization separately from the in-distribution viewpoint split.","section":"§4.3"},{"comment":"Section 4.1 and Table 1 advertise a four-model benchmark (UNet-no-skip, ResNet-50, Swin-T, ConvNeXt-T), but Tables 4 and 6 report results only for ResNet-50 and Swin-T. The paper does not explain why the other two models are missing. As written, the claim of a 'four-model benchmark' is unsupported by the experimental section.","section":"§4.1–4.2 and Tables 4, 6"},{"comment":"Section 5 states that MATPREDICT is 'the first dataset that factorises material diversity from object geometry'. The dataset construction does pair each object with multiple materials, but the benchmark never varies geometry and material independently at test time; no experiment holds geometry fixed while changing material, or vice versa. The novelty claim should be reworded to describe the dataset's construction, not the demonstrated learning result, until such experiments are provided.","section":"§5"}],"minor_comments":[{"comment":"Table 6, Concrete row: '0.0.0021' contains a typo and should presumably read '0.0021'.","section":"Table 6"},{"comment":"The sentence 'we shuffle S once and split it 80 SUBSET' is incomplete; specify the exact train/validation/test proportions and how the validation set is used.","section":"§4.3"},{"comment":"Table 5 gives the SAM range as [0, π/2] rad, while Tables 4 and 6 report SAM in degrees; please be consistent about angular units.","section":"Appendix A, Table 5"},{"comment":"The text reads 'MatSynth contains from than 4000' and should read 'contains more than 4000'.","section":"§3"},{"comment":"Table 3 is labelled 'partial' but no full 18×14 matrix is supplied; either provide the complete matrix or state clearly how the six shown rows were selected.","section":"Table 3"},{"comment":"The code fragment 'we keep–if_cropped False' should be replaced by a prose description of the cropping behaviour.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The SAM inconsistency is severe enough that the paper should not be accepted in its current form. The dataset release and pipeline description are valuable, and the circularity concern can be addressed with additional experiments, so I recommend major revision rather than rejection. I would also ask the editor to verify that the dataset links are accessible, since the paper's main contribution depends on them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the dataset artifact is real and worth having; the benchmark evidence as printed is not. The SAM values contradict the other metrics, the evaluation split lets the networks cheat by memorizing per-pair textures, and only two of the four advertised models are actually reported.\n\nWhat's new: MatPredict combines Replica object meshes with MatSynth materials through a Blender Cycles pipeline, with explicit UV texel-density normalization, UDIM packing, a stratified camera shell, and a reproducible lighting rig. That specific combination isn't in SUNCG, PBRS, CGIntrinsics, or CG-PBR, and they cite those predecessors. The data and code links are real, and the pipeline description is detailed enough to reproduce. The limitations appendix is refreshingly honest: they admit the synthetic-to-real gap, the single-material-per-object restriction, and that only two material layers are benchmarked. That counts for something.\n\nSoft spots, in order of severity:\n\n1. The metric inconsistency is glaring. In Table 4, SAM values sit around 87-90 degrees while RMSE is below 0.007 and SSIM above 0.94. For near-identical images, SAM should be near zero degrees; 87-90 degrees means the predicted and ground-truth color vectors are nearly orthogonal. These numbers cannot both be right as printed. Someone needs to recompute those columns.\n\n2. The benchmark split doesn't support the generalization claim. The 512 screenshots of each object-material pair are shuffled and split 80/20 by seed, so every test image shares its object and material with training images. The ground-truth basecolor and roughness maps are exactly the maps used to render those pairs. The network can identify the pair from coarse shape and emit a memorized texture map, which would give exactly the low RMSE and high SSIM reported. There is no cross-object, cross-material, or real-image experiment, so the Section 5 claim that MatPredict 'factorises material diversity from object geometry' is untested.\n\n3. The paper advertises a four-model benchmark but reports only ResNet-50 and Swin-T in Tables 4 and 6. Where are UNet-no-skip and ConvNeXt-T? That's a missing-results issue, not a style nit.\n\nThese are fixable. The dataset itself doesn't depend on the benchmark being correct. A leave-one-object-out split, or at least an explicit statement that the task is instance-specific reconstruction rather than generalization, would make the conclusions honest. The SAM columns need to be recomputed or explained.\n\nWho this is for: robotics and inverse-rendering researchers who want a small, controllable synthetic corpus with known material maps. It's a data paper, not an algorithmic breakthrough. As is, I would not accept the benchmark at face value. But the dataset deserves a serious referee: it may be a useful resource, and the problems are correctable. I'd send it to review with a request for major revision, and I'd want to see the full result tables and a generalization experiment before signing off.\n\nI would not cite it for the benchmark numbers yet, but I might cite it as a dataset resource once it's cleaned up.","headline":"A genuinely new synthetic dataset for object-material appearance, but the benchmark as printed can't support the generalization claim; worth a conditional peer review.","tokens_in":9664,"tokens_out":3453,"would_cite":false,"duration_ms":31733,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MatPredict renders 18 indoor objects in 14 materials and reports that standard neural networks recover their basecolor and roughness maps from a single image.","keywords":["material property prediction","physically based rendering","inverse rendering","synthetic dataset","indoor robotics","basecolor estimation","roughness estimation","image regression"],"falsifier":"Train on a subset of object identities and test on the remaining identities rendered with the same materials; if per-material error rises sharply for unseen shapes while staying low for seen ones, the reported recovery is shape memorization, not material inference.","tokens_in":8729,"feed_emoji":"🤖","tokens_out":12114,"duration_ms":117271,"temperature":0.7,"pith_summary":"MatPredict is a synthetic dataset and benchmark for teaching machines to infer visual material properties from a single image. The paper pairs indoor object meshes with a library of physically based materials, renders 18 common objects in 14 materials under varied lighting and viewpoints, and reports that four encoder-decoder models can recover pixel-wise basecolor and roughness maps from a $224\\times224$ crop. The motivation is robotic manipulation: knowing whether a mug is ceramic or metal tells a robot how hard to grip it. The paper's contribution is decoupling material diversity from object geometry, so material recognition can be studied without the usual shape-material confound.","feed_headline":"18 objects, 14 materials: new benchmark for material recognition","feed_subtitle":"In the MatPredict benchmark, four standard networks regress basecolor and roughness from a single 224-by-224 crop.","key_machinery":"The load-bearing mechanism is the physically based rendering pipeline that makes the image a deterministic function of the material maps. A mesh is paired with a material stack through a Principled BSDF shader (a single node bundling basecolor, roughness, metallic, and related parameters), after a UV pass rescales texel density to object size and packs the UV islands into UDIM tiles to prevent seams. A spherical camera grid and a symmetric ring of area lights produce 512 views per object-material pair. On the regression side, all four networks share an encoder-decoder design whose only material-dependent component is a final $1\\times1$ convolution, so basecolor and roughness are predicted jointly and new material channels can be added by changing the output kernel count.","core_discovery":"The central discovery is that a controlled synthetic pipeline can separate material appearance from object shape well enough for current networks to recover the underlying reflectance maps. Each object mesh is paired with one of 14 material stacks from a PBR library and rendered from 512 viewpoints on a fixed spherical grid, after a UV pass equalizes texel density so a teacup and a wardrobe show comparable detail. Benchmarked UNet, ResNet-50, Swin-T, and ConvNeXt-T models regress a six-channel target of basecolor plus roughness; the best combinations reach RMSE around 0.001 and SSIM above 0.99. The paper presents MatPredict as the first dataset that factorises material diversity from object geometry.","pith_inferences":["A stronger test the paper does not run is holding out entire object identities or material classes; if error jumps for unseen shapes, low viewpoint-holdout error is partly texture memorization.","The reported spectral-angle errors stay near the metric's upper bound even where RMSE is tiny, so per-material and per-channel SAM analysis would clarify whether networks match brightness but miss spectral direction.","The current renders assign one uniform material per object; composing several materials on one mesh, as the paper lists as future work, would let the same pipeline support objects like a wooden table with metal legs.","Because the ground-truth maps come from the same renderer as the images, the benchmark scores reflect simulator consistency; evaluating on a small set of real photographs would test transfer to physical cameras."],"forward_implications":["Because geometry and material vary independently in the dataset, robotic perception systems can train material recognition without the confound present in real photographs.","The four models, trained with identical splits and schedules, give the community a common baseline for comparing future material-regression methods.","Extending the task to metallicity, normal maps, transparency, or other material layers requires only changing the number of output channels in the final $1\\times1$ convolution and re-weighting the loss per channel.","The deterministic camera grid and lighting rig mean the dataset can be regenerated or extended at different densities without changing the evaluation protocol."],"supporting_citations":[{"why":"supplies the photorealistic indoor scene meshes from which individual object sub-meshes are extracted.","marker":"[20]"},{"why":"supplies the PBR material property sets, including basecolor and roughness maps, that are swapped onto the objects.","marker":"[23]"},{"why":"provides the open-source renderer and Principled BSDF shader used to generate the images.","marker":"[6]"},{"why":"provides the semantic segmentation step that isolates each object into a crop for the benchmark.","marker":"[11]"},{"why":"provides the ResNet-50 backbone evaluated as one of the four benchmark models.","marker":"[9]"},{"why":"provides the Swin-T transformer backbone evaluated in the benchmark.","marker":"[14]"},{"why":"provides the ConvNeXt-T backbone evaluated in the benchmark.","marker":"[15]"},{"why":"provides the UNet architecture used as the compact baseline model.","marker":"[16]"}],"fun_headline_variants":["MatPredict: 18 objects, 14 materials, 512 viewpoints","New benchmark separates material from shape in synthetic renders","Material properties inferred with RMSE under 0.002 from images","MatPredict: first dataset to factor material diversity from geometry","Synthetic pipeline enables accurate material recognition for robotics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that holding out viewpoints of the same object-material pairs is enough to prove material learning, so low test error can just mean the network memorized each object's textures rather than learning to infer materials from appearance.","fun_headline_variants_meta":{"raw":{"variants":["MatPredict: 18 objects, 14 materials, 512 viewpoints","New benchmark separates material from shape in synthetic renders","Material properties inferred with RMSE under 0.002 from images","MatPredict: first dataset to factor material diversity from geometry","Synthetic pipeline enables accurate material recognition for robotics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001486,"raw_usage":{"total_tokens":5952,"prompt_tokens":915,"completion_tokens":5037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":4955}},"tokens_in":531,"tokens_out":5037,"duration_ms":38688,"temperature":1.0,"reasoning_tokens":4955,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:17:17.798377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on a subset of object identities and test on the remaining identities rendered with the same materials; if per-material error rises sharply for unseen shapes while staying low for seen ones, the reported recovery is shape memorization, not material inference.","supporting_citations":[{"cited_title":"MatSynth: A Modern PBR Materials Dataset","cited_arxiv_id":"2401.06056","evidence_quote":"supplies the PBR material property sets, including basecolor and roughness maps, that are swapped onto the objects."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"provides the Swin-T transformer backbone evaluated in the benchmark."},{"cited_title":"A convnet for the 2020s","cited_arxiv_id":null,"evidence_quote":"provides the ConvNeXt-T backbone evaluated in the benchmark."}],"review_version":1}