{"id":"2bcc6074-6550-4a75-8545-8dc16f826990","arxiv_id":"2508.02131","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A neural network predicts perceived quality of BRDF approximations directly from sampled BRDF values, reporting higher correlation with human ratings than traditional BRDF-space metrics.","lead":"This paper trains a small neural network to score how similar two reflectance functions (BRDFs) are to human perception, without rendering images. The metric outperforms classic BRDF-space metrics on a small test set, but the evaluation setup may inflate its reported advantage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.67 Spearman correlation is not independent of the test set: the ΔE_ITP-to-JOD calibration (Eq. 1) is fitted on the same 20 human-rated materials later used for evaluation, so the neural metric inherits a mapping already optimized to those test judgments.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the ΔE_ITP-to-JOD mapping in Eq. 1 is fitted on the same 20 materials later used for testing, so the test set is not independent of the label-generation process. I traced this through the paper: Section 3.1 estimates b1, b2, b3 on the perceptual-experiment subset, Section 3.1 then uses those parameters to generate pseudo-labels for all materials, and Section 4 reserves those same perceptual-experiment BRDFs for testing. The network never directly sees human JODs during training, but its training targets are produced by a function calibrated on the test materials' human judgments. Consequently, the reported Spearman correlation of 0.67 can be inflated even if the network generalizes perfectly to the pseudo-label function: that function already encodes the test set's human responses. The comparison to uncalibrated BRDF-space metrics is therefore not apples-to-apples. The proposed leave-one-material-out check would directly quantify the inflation by re-fitting Eq. 1 on training-only materials and evaluating on a held-out human-rated material. The concern is severe enough that the current version's central claim is unsupported, matching the reader's REJECT verdict. I do not see a separate, more load-bearing issue: the network architecture, training, and code availability are all reasonable, but they cannot rescue an evaluation protocol that leaks test information into the label construction.","tokens_in":7178,"tokens_out":4110,"duration_ms":53678,"concrete_test":"Run leave-one-material-out validation over the 20 materials with human JODs. For each held-out material m, (i) fit b1, b2, and b3 in Eq. 1 using only the ΔE_ITP and human-JOD pairs from the other 19 materials; (ii) generate pseudo-labels for all non-test BRDFs with those parameters; (iii) train BRDF-NQM with the same architecture and hyperparameters; and (iv) compute the Spearman correlation between its predictions and the human JODs for the held-out material m. Average the correlation across all 20 folds and compare it with the 0.67 value in Figure 3. If the average drops toward the 0.38–0.55 baseline range, the reported advantage is an artifact of fitting the label-mapping parameters on the test set; if it remains near 0.67, the central claim survives this check.","verdict_should_be":"REJECT","load_bearing_attack":"For the central claim to hold, BRDF-NQM's predictions on the 180 test instances must be compared with human JODs in a way that is independent of how the training targets were constructed. Section 3.1 states that the parameters b1, b2, and b3 in Eq. 1 are estimated using Levenberg–Marquardt on 'the subset of materials used in the perceptual experiment'—the same 20 materials that Section 4 says are 'reserved for testing.' Those parameters are then used to compute JOD pseudo-labels for all materials, including the training data. The network is therefore trained to predict a calibrated function f(ΔE_ITP) whose parameters were fitted to human judgments on the exact materials that constitute the test set. At evaluation time, BRDF-NQM's outputs on held-out distorted instances of those 20 materials are approximately f(ΔE_ITP), and f was specifically chosen to minimize error against the human JODs on that same set. The reported average Spearman correlation of 0.67 is thus not an independent measure of perceptual alignment; it partly reflects the calibration curve's fit to the test responses. Comparing this calibrated neural metric against raw, uncalibrated BRDF-space baselines is not a fair test of the claimed 'significantly higher correlation.' The paper honestly discloses the pseudo-labeling procedure but does not address this leakage in the evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BRDF-NQM, a compact multi-layer perceptron that predicts a just-objectionable-difference (JOD) score directly from 500 paired samples of a reference and an approximated BRDF, without rendering. The network is trained on pseudo-labels produced by a calibrated image-space metric, ΔE_ITP, applied to a dataset of 159 measured BRDFs and augmented with noisy BRDF pairs. The authors report that BRDF-NQM achieves an average Spearman correlation of 0.67 with human perceptual judgments on a 20-material test subset, outperforming eight BRDF-space baseline metrics. They also report that using the metric as a loss function for BRDF fitting yields limited results due to color shifts.","tokens_in":7475,"tokens_out":6201,"duration_ms":75151,"significance":"The underlying idea—a rendering-free, neural quality metric operating directly in BRDF space—is potentially useful for BRDF fitting, model selection, and compression, and the paper includes an open-source implementation and a compact architecture with fast inference. However, the central empirical claim of substantially improved correlation with human judgments is compromised by the evaluation protocol: the calibration function used to generate training labels is fitted on the same subjective data that later constitutes the test set. As presented, the reported 0.67 Spearman correlation does not establish an independent perceptual alignment, and the comparison against uncalibrated baselines is not like-for-like. The concept is interesting, but the validation needs to be substantially reworked.","major_comments":[{"comment":"The calibration of ΔE_ITP to JOD values uses the Levenberg–Marquardt algorithm on \"the subset of materials used in the perceptual experiment\"—the same 20 materials that Section 3.1 later reserves for testing. The fitted parameters (b1, b2, b3) in Eq. (1) are then used to compute pseudo-labels for all training data, and at test time the network's predictions on the 180 held-out instances are effectively approximations of this calibrated function. The reported Spearman correlation of 0.67 therefore partly reflects the goodness of fit of the calibration curve to the test responses, not an independent measure of perceptual alignment. To support the claim, the authors must use a disjoint split for calibration and testing (e.g., leave-one-out or k-fold over the 20 materials), and should also calibrate the baseline metrics with the same regression procedure to ensure a fair comparison.","section":"Section 3.1, Eq. (1) and Section 4"},{"comment":"The paper does not specify how JOD pseudo-labels are assigned to the 3,340 augmented BRDF pairs generated by adding Gaussian noise with σ=0.01. If ΔE_ITP values for these pairs are computed by rendering the noisy BRDFs, the rendering pipeline and its cost should be described; if instead the JOD values are inferred without rendering, it is unclear how Eq. (1) applies to noise-perturbed BRDF samples. This information is necessary for reproducibility and for assessing the validity of the training targets.","section":"Section 3.1, augmentation paragraph"},{"comment":"The paper reports only a single average Spearman correlation (0.67) without error bars, confidence intervals, or a significance test against the best baseline (0.55). Given the small test set (180 instances from 20 materials), the claim of \"significantly higher correlation\" requires a statistical analysis, such as a permutation test or bootstrapping across materials, to establish that the improvement is not within sampling variability.","section":"Section 4, Figure 3"}],"minor_comments":[{"comment":"The expression includes `max(ΔE_ITP, 0)^b3` with a negative exponent b3 = −0.21; for ΔE_ITP = 0 this term is undefined. Please clarify the domain and any numerical safeguards used.","section":"Equation (1)"},{"comment":"The phrase \"perceptually validated image-space metric\" is vague; specify that ΔE_ITP was selected based on its previously reported correlation with human judgments.","section":"Abstract"},{"comment":"The description of selecting 500 BRDF samples says samples are chosen based on reflectance magnitude while also using uniform angular sampling; please clarify how these two criteria are combined.","section":"Section 3.1, sampling strategy"},{"comment":"The order of baseline metrics in the figure legend is not consistently aligned with the text in Section 4; ensure the labels match the described definitions.","section":"Figure 3"},{"comment":"There are typographical errors in author names (e.g., \"Rafa l\" should be \"Rafał\") and in the acknowledgments (\"Sk lodowska\"); these should be corrected in the final version.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The circular evaluation is the main barrier to publication. If the authors can re-run the experiments with a clean calibration/test split and include statistical significance testing, the paper could become a solid contribution. I would not consider acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about arXiv:2508.02131 is that its central claim—that BRDF-NQM beats BRDF-space metrics with an average Spearman correlation of 0.67—does not survive contact with the evaluation protocol. The pseudo-label mapping in Eq. (1) is fitted with Levenberg–Marquardt on the same 20 materials that are later reserved for testing. The network is trained to approximate that calibrated function, so the test scores partly measure how well the curve was fitted to those human judgments, not an independent prediction. The stress-test note is correct.\n\nWhat the paper does well: it is the first to apply a neural metric directly in BRDF space for quality assessment, and the idea is sensible—learn a rendering-free approximation to an image-space perceptual metric. The MLP architecture is simple but adequate, and the authors ship code. They also honestly report that the metric fails as a loss function, which is a real negative result. The writing is clear and the related work is appropriately cited.\n\nThe soft spots, in proportion: the circularity is load-bearing, not a minor artifact. The test set is only 180 distorted instances across 20 materials, and there are no confidence intervals, so the gap between 0.55 and 0.67 could well be within noise. The augmentation with uncorrelated noise and the whitening step are plausible but not carefully ablated. That said, the paper's own Section 3.1 discloses the pseudo-labeling procedure, so this is not hidden misconduct—it is a methodological oversight that should have been caught before submission.\n\nIf the authors re-run the calibration on a disjoint set of materials and report confidence intervals, the result could be credible. As is, the paper is not publishable in this form. I would send it to peer review because the idea is worth refereeing and the fix is straightforward, but the decision would be reject or major revision. For your reading group: maybe worth a slot to discuss evaluation leakage, but not as a model to build on.","headline":"The 0.67 correlation is inflated because the JOD calibration is fitted on the test materials, so the central claim doesn't hold, though the idea is worth a revised attempt.","tokens_in":8013,"tokens_out":1715,"would_cite":false,"duration_ms":21892,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes BRDF-NQM, a compact neural network that predicts the perceptual quality of BRDF approximations directly from BRDF samples, without rendering, and reports an average Spearman correlation of 0.67 with human judgments.","keywords":["BRDF quality metric","perceptual quality","neural network","just-objectionable-difference","material appearance","BRDF fitting","image-space metric","Rusinkiewicz coordinates"],"falsifier":"Retrain or re-evaluate with strict material separation: fit the $\\Delta E_{\\mathrm{ITP}}$-to-JOD parameters on one set of materials and test on disjoint materials with human ratings. If the Spearman correlation drops toward the 0.4 to 0.6 baseline range, the reported 0.67 advantage would be shown to depend on the label-fitting overlap rather than on a generalizable perceptual mapping.","tokens_in":1644,"feed_emoji":"🎨","tokens_out":2024,"duration_ms":71725,"temperature":0.7,"pith_summary":"This paper introduces BRDF-NQM, a compact neural network that predicts how much a fitted BRDF model will deviate perceptually from the reference material, using only samples of the two BRDFs rather than rendered images. The authors train a four-layer MLP on paired reference and approximated BRDFs from measured datasets, using just-objectionable-difference (JOD) scores as targets; for most of the data the JODs are pseudo-labels obtained by fitting $\\Delta E_{\\mathrm{ITP}}$ color-difference values to the perceptual scores from an earlier study. On the 180 test instances with human ground truth, the metric reaches an average Spearman correlation of 0.67 with subjective judgments, clearly above the 0.38 to 0.55 range of traditional BRDF-space error metrics. The practical payoff is a quality evaluation that runs in milliseconds and needs no rendering, though the authors report that using the same network as a loss for BRDF fitting still produces color artifacts.","feed_headline":"Neural metric scores BRDF model quality, no rendering needed","feed_subtitle":"A compact MLP trained on measured BRDFs agrees with human ratings at 0.67 Spearman, far above traditional error metrics.","key_machinery":"The load-bearing object is the learned mapping from BRDF samples to perceived quality, implemented as a four-layer MLP with layer normalization, GELU activations, dropout, and a sigmoid-scaled output. Its inputs are 500 reflectance samples per BRDF, paired as reference and approximation, selected in the Rusinkiewicz coordinate system with denser sampling near the half-vector origin to capture specular structure. The training targets come from a fitted function $\\mathrm{JOD} = f(\\Delta E_{\\mathrm{ITP}})$ with parameters $b_1, b_2, b_3$ estimated on the perceptual-study materials, making the image-space metric $\\Delta E_{\\mathrm{ITP}}$ the teacher that transfers human judgments into BRDF-space labels. This combination of perceptual pseudo-labels, a subsampled angular representation, and a small MLP is what lets the metric run in 0.04 ms per prediction without rendering.","core_discovery":"The central claim is that the perceptual quality of a BRDF approximation can be learned and predicted directly in BRDF space. BRDF-NQM takes 500 paired samples of reference and approximated BRDFs, transformed to a cube-root, logarithmic, whitened representation in Rusinkiewicz coordinates, and outputs a scalar JOD value. Trained with pseudo-labels derived from the perceptually validated $\\Delta E_{\\mathrm{ITP}}$ metric, the network achieves the highest correlation with human judgments among all compared BRDF-space metrics, averaging 0.67 Spearman versus a top baseline of 0.55. The paper also reports that using BRDF-NQM as a loss function for BRDF fitting yields partially accurate parameters but introduces a red color shift in the estimated diffuse component, so predictive quality and optimization-loss quality do not automatically coincide.","pith_inferences":["Because the test materials overlap the materials used to fit the label-generation curve, the headline 0.67 is best read as an upper bound; a fully held-out material evaluation would likely give a lower, more realistic generalization estimate.","The same architecture could be adapted to other appearance representations, such as SVBRDF or BTF, by replacing the sampling scheme, provided perceptual pseudo-labels are available.","If the training-data color imbalance is corrected and the whitening step is made more chromatic-sensitive, the observed red shift as a loss function might be mitigated, turning the metric into a gradient source for perceptually guided fitting.","A testable extension would be to use angularly correlated noise for low-JOD augmentation, which could improve sensitivity to structured distortions instead of per-direction independent noise."],"forward_implications":["BRDF fit quality can be checked in real time (0.04 ms per prediction) without rendering, making perceptually informed evaluation practical in iterative workflows.","Traditional BRDF-space metrics such as RMSE, MAE, and their cube-root or logarithmic variants are outperformed in correlation with human judgment, suggesting representation-space error alone is not enough.","The same learned quality signal does not yet serve as a reliable loss function, since fitting with it produced a red color shift in the diffuse component; predictive accuracy does not guarantee optimization usefulness.","The success of transferring image-space perceptual labels into BRDF space suggests future BRDF-space loss functions can be designed from perceptual data rather than from ad hoc distance measures."],"supporting_citations":[{"why":"Supplies the dataset of 159 BRDFs, 1,431 distorted videos, the subjective JOD ratings, and the evaluation of image-space metrics such as Delta-E_ITP.","marker":"[10]"},{"why":"Provides the MERL measured BRDF dataset that is a primary source of material samples.","marker":"[19]"},{"why":"Provides the RGL-EPFL measured BRDF dataset used in the training data.","marker":"[5]"},{"why":"Provides the DTU measured BRDF dataset used in the training data.","marker":"[22]"},{"why":"Defines the Rusinkiewicz coordinate system used for subsampling the BRDF angular domain.","marker":"[23]"},{"why":"Motivates the cube-root transformation and contributes a perceptual quality dataset for BRDF approximations.","marker":"[13]"},{"why":"Prior work showing that image-space metrics align better with perception and that metric accuracy does not guarantee performance as a fitting loss.","marker":"[2]"},{"why":"The Levenberg-Marquardt algorithm used to fit the Delta-E_ITP-to-JOD regression parameters.","marker":"[14]"},{"why":"The Marquardt variant of the least-squares algorithm used in the parameter fitting.","marker":"[18]"}],"fun_headline_variants":["First neural BRDF quality metric skips rendering","BRDF quality scored by compact MLP, no rendering","Neural metric predicts perceived BRDF quality directly","Perceptual BRDF metric learned, not rendered"],"cache_read_input_tokens":10112,"weakest_assumption_plain":"The load-bearing premise is that the JOD pseudo-labels are valid for every material, even though the $\\Delta E_{\\mathrm{ITP}}$-to-JOD function is fitted on the same 20 BRDF materials that are later used as the test set, so the test labels are not independent of that fitting step.","fun_headline_variants_meta":{"raw":{"variants":["First neural BRDF quality metric skips rendering","BRDF quality scored by compact MLP, no rendering","Neural metric predicts perceived BRDF quality directly","Perceptual BRDF metric learned, not rendered"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2648,"prompt_tokens":885,"completion_tokens":1763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":1702}},"tokens_in":501,"tokens_out":1763,"duration_ms":14166,"temperature":1.0,"reasoning_tokens":1702,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:07:33.686203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain or re-evaluate with strict material separation: fit the $\\Delta E_{\\mathrm{ITP}}$-to-JOD parameters on one set of materials and test on disjoint materials with human ratings. If the Spearman correlation drops toward the 0.4 to 0.6 baseline range, the reported 0.67 advantage would be shown to depend on the label-fitting overlap rather than on a generalizable perceptual mapping.","supporting_citations":[{"cited_title":"Perceived Quality of BRDF Models","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 159 BRDFs, 1,431 distorted videos, the subjective JOD ratings, and the evaluation of image-space metrics such as Delta-E_ITP."},{"cited_title":"A Data-Driven Reflectance Model","cited_arxiv_id":null,"evidence_quote":"Provides the MERL measured BRDF dataset that is a primary source of material samples."},{"cited_title":"An Adaptive Parameterization for Efficient Material Acqui- sition and Rendering","cited_arxiv_id":null,"evidence_quote":"Provides the RGL-EPFL measured BRDF dataset used in the training data."},{"cited_title":"On Optimal, Minimal BRDF Sampling for Reflectance Acquisition","cited_arxiv_id":null,"evidence_quote":"Provides the DTU measured BRDF dataset used in the training data."},{"cited_title":"A New Change of Variables for Efficient BRDF Representation","cited_arxiv_id":null,"evidence_quote":"Defines the Rusinkiewicz coordinate system used for subsampling the BRDF angular domain."},{"cited_title":"Perceptual quality of BRDF approximations: dataset and metrics","cited_arxiv_id":null,"evidence_quote":"Motivates the cube-root transformation and contributes a perceptual quality dataset for BRDF approximations."},{"cited_title":"An Adaptive BRDF Fitting Metric","cited_arxiv_id":null,"evidence_quote":"Prior work showing that image-space metrics align better with perception and that metric accuracy does not guarantee performance as a fitting loss."},{"cited_title":"A method for the solution of certain nonlinear problems in least squares","cited_arxiv_id":null,"evidence_quote":"The Levenberg-Marquardt algorithm used to fit the Delta-E_ITP-to-JOD regression parameters."},{"cited_title":"An algorithm for least-squares estimation of nonlinear parameters","cited_arxiv_id":null,"evidence_quote":"The Marquardt variant of the least-squares algorithm used in the parameter fitting."}],"review_version":1}