Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

The paper claims that OpenVTON-Bench, a 100K-pair high-resolution benchmark with a hybrid evaluation metric, agrees with human preference better than standard pixel metrics, and that its erosion-based profile separates boundary alignment er

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:28 UTC pith:RG3XJSYD

load-bearing objection A large, well-intentioned VTON benchmark whose headline metric ranking is not supported by the paper's own correlation tables. the 4 major comments →

arxiv 2601.22725 v4 pith:RG3XJSYD submitted 2026-01-30 cs.CV cs.AI

OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

Jin Li , Tao Chen , Kai Wen , Siqi Yin , Shuai Jiang , Weijie Wang , Jingwen Luo , Chenhui Wu This is my paper
classification cs.CV cs.AI
keywords virtual try-on evaluationbenchmark datasethigh-resolution image pairsimage quality assessmentmulti-scale representation metricmorphological erosionvision-language model judgetexture fidelity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

OpenVTON-Bench is an attempt to close the gap between how rapidly virtual try-on models are improving and how reliably they can be judged. The paper assembles roughly 100,000 high-resolution paired images (up to 1536x1536) in which a standalone garment image maps to a real photo of a person wearing it, balanced across 20 garment categories, and supplements each pair with dense captions and occlusion masks. On top of this dataset, it proposes an evaluation protocol that scores generated try-ons on five dimensions — background consistency, identity, texture, shape, and overall realism — by combining a vision-language judge with a representation metric computed on progressively eroded garment masks. The authors report that this hybrid metric agrees with human preference better than standard pixel-based metrics, with the multi-scale representation score reaching a pairwise ranking correlation of 0.833 versus 0.611 for SSIM. If the ground truth pairs are reliable, this gives the field a large open testbed and a diagnostic tool that can separate boundary-alignment errors from interior texture errors.

Core claim

The paper's central claim is that virtual try-on quality can be measured along five interpretable dimensions — background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism — and that a hybrid of a vision-language model scoring and a multi-scale feature-similarity score tracks human preference more closely than conventional pixel-level metrics. The load-bearing technical idea is to mask the garment, progressively erode that mask so that boundaries are excluded, and compute feature cosine similarities inside the remaining interior regions at each erosion level. Averaging those interior scores, plus a masked global similarity, gives the benchmark's repres

What carries the argument

The key mechanism is the Multi-Scale Representation Metric: a garment is isolated by a segmentation model's mask, then that mask is eroded repeatedly with a small structuring element to produce nested regions; at each erosion level, cosine similarity is computed between self-supervised image features of the generated image and the ground-truth image restricted to that masked region. Early levels cover boundaries, deeper levels expose only interior fabric, so the profile across levels separates whether a model fails at edge alignment or at reproducing patterns, logos, and materials. A separate vision-language model scores the same five semantic dimensions, and the paper averages the multi-sca

Load-bearing premise

The load-bearing premise is that the 99,925 garment-to-model pairings and their occlusion masks are correct; if a non-trivial fraction are mismatched or the masks do not align with the actually worn garment, then every ground-truth comparison, every multi-scale score, and every model ranking in the paper is compromised.

What would settle it

Audit the ground truth directly: take a random sample of 200 triplets, have fresh annotators draw the worn garment's mask and judge whether the garment image matches the worn item, and compute mean IoU between the benchmark masks and the independent masks. If the mismatch rate is high or the IoU is poor, the claimed human correlation of the multi-scale metric cannot be accepted as evidence about model quality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • With roughly 100,000 paired, high-resolution samples and open release, the benchmark gives the community a standard testbed for training and comparing virtual try-on systems under realistic e-commerce conditions.
  • The five-dimension protocol turns evaluation into diagnostics: a model that passes realism but fails texture can be identified, rather than hidden in an aggregate score.
  • Because the erosion curve separates boundary from interior errors, benchmark users can decide whether to invest in warping and alignment fixes or in texture-preservation training.
  • The reported gap between realism and texture scores implies that large general-purpose generative models need domain-specific fine-tuning for garment fidelity, not just bigger backbones.
  • The correlation analysis offers a template for future virtual try-on papers to report human agreement, not just pixel errors.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not pursue: apply the erosion-sweep method to other paired editing tasks — face editing, relighting, object replacement — where edge alignment and interior content are separable; if the same correlation pattern holds, the metric is a general diagnostic.
  • The deep-erosion scores use ground-truth masks; one could instead compute them against the input garment image alone, potentially yielding a reference-free texture fidelity measure for tasks where paired ground truth exists only at training time.
  • The paper's own limitation note that foundation-model captions may inherit biases is worth taking seriously: if the dense captions drive any filtering, the 'balanced' category structure could silently reflect the captioner's blind spots, so an independent audit of a sample is a cheap safeguard.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces OpenVTON-Bench, a benchmark of approximately 99,925 high-resolution (up to 1536×1536) garment-model pairs, constructed with DINOv3-based semantic clustering for category balance and Gemini-generated dense captions. It also proposes a hybrid evaluation protocol that combines a VLM-as-a-Judge (Qwen-VL-Plus) with a representation-based metric built on DINOv3 features, SAM3 masks, and multi-scale morphological erosion, with the goal of separating boundary misalignment from internal texture errors. The central claim is that the proposed Representation Metric outperforms traditional metrics in correlating with human judgments, reporting Kendall's tau of 0.833 versus 0.611 for SSIM, based on a nine-method meta-evaluation.

Significance. If the claims are substantiated, the benchmark and protocol would be a useful public resource: it is large-scale, high-resolution, open-source, paired, and more diverse than prior studio-centric VTON datasets, and the decomposition into five interpretable dimensions is practical for diagnosing model failures. The human study, with over 90,000 ratings, is a substantial effort. However, the headline metric is not consistently defined across the paper, the reported correlations lack uncertainty quantification, and key dataset-quality guarantees are not demonstrated. Because these issues directly affect the benchmark's central claims, the paper requires major revision before the conclusions can be accepted.

major comments (4)
  1. [Sec. 3.2.2, App. D.2, Table 3, Table 5, Table 7] The metric that is claimed to dominate is not consistently defined. Eq. 6 defines S_rep^{(k)}, and App. D.2 states that the reported \bar S_rep is the arithmetic mean over k=0..3. Table 7 reports rho_s=0.850 and rho_k=0.667 for every S_rep^{(k)} and for \bar S_rep. Table 5, however, reports rho_s=0.933 and rho_k=0.833 for '\bar S(Rep)'. These numbers cannot both refer to \bar S_rep. Inspection of Table 3 shows the final column, \bar S, equals (S_global + \bar S_rep)/2 up to rounding, so the headline metric is a composite of the global cosine similarity and the eroded-mask average, not the multi-scale representation metric alone. The paper never defines \bar S in Sec. 3.2.2. Please define the composite explicitly, relabel Table 5 consistently with Table 7, and re-state the abstract and Sec. 4.3 claims accordingly. As written, the abstract's 'Kendall's tau of 0.833' is attributed to the wr
  2. [Table 5, Table 7, Sec. 4.3] All correlations are computed over nine method-level aggregates, i.e., one point per method. With n=9, Spearman and Kendall are extremely noisy; Kendall's tau can take only 36 discrete values, so the difference between 0.833 and 0.667 corresponds to roughly six discordant pairs out of 36. The paper reports no confidence intervals, bootstrap, permutation tests, or significance tests for any of the correlation coefficients. This is load-bearing because the central claim is that the proposed metric dominates SSIM/LPIPS/FID in ranking models; those rank differences may be well within sampling noise. Please provide per-dimension or per-sample correlations, bootstrap confidence intervals, and paired permutation tests for the rankings.
  3. [Sec. 2.2, App. A.1, App. A.3, Limitations] The benchmark's ground truth is load-bearing. The paper states that over 1,000 annotators verified that the garment strictly matches the model's outfit, but it reports no inter-annotator agreement, no error rate after verification, no protocol for handling disagreement, and no quality-control statistics on the SAM3/GroundingDINO masks used to create occlusion masks and compute S_rep. The Limitations section explicitly concedes that foundation-model annotations may inherit semantic biases or hallucinations. Without a quantitative audit of pair mismatches and mask errors, the validity of every downstream comparison is unclear. Please report a QA audit on a random sample, including human re-verification of pairs and mask overlap/accuracy statistics.
  4. [Sec. 2.3 vs. Sec. 3.2.2] DINOv3 is used both to construct the balanced dataset (hierarchical clustering and stratified sampling in Sec. 2.3, App. A.2) and to define the representation metric (Eqs. 3-6). If the benchmark is explicitly selected to be separable in DINOv3 feature space, a DINOv3-based metric may be favorably conditioned for this dataset, potentially inflating its correlation with human judgments relative to a metric using a different feature space. This is not a deductive flaw because the human ratings are independent, but it is a correctness risk. Please report the same meta-evaluation using alternative frozen features (e.g., DINOv2, CLIP) or on a random subset of the candidate pool to show that the ranking result is not an artifact of using the same representation for selection and evaluation.
minor comments (6)
  1. [Sec. 2.3 vs. App. A.2] The main text says 'hierarchical clustering', while the appendix describes 'K-Means clustering' of DINOv3 embeddings. Please harmonize the description.
  2. [App. B.2] The appendix states that the representation metric computes cosine similarity between the Reference Garment and the Generated Image, while Eq. 6 in the main text compares the Generated Image with the Ground Truth. Please correct the appendix.
  3. [Table 2] There are formatting errors in the table, e.g., '4.2534.689' and '4.0504.018', where numbers are concatenated without spacing. Please fix the LaTeX/table rendering.
  4. [Sec. 3.3 vs. App. D.3] The main text says 'over 90,000 valid ratings', while the appendix reports '92,072 samples'. Clarify whether these are ratings, triplets, or per-dimension scores, and specify how the 76 participants were assigned to items.
  5. [Sec. 6 / Data Release] The paper says 'Code and dataset are available here' but no URL or repository is provided. This makes the claimed open-source contribution unverifiable. Please include a working link.
  6. [Table 3] The final column \bar S is introduced without a definition in the main text or table caption. Add a formula or a reference to Sec. 3.2.2 once the composite is defined.

Circularity Check

0 steps flagged

No significant circularity: the metric is evaluated against an independent human study and is not fitted to its own inputs.

full rationale

OpenVTON-Bench's central derivation is the proposed evaluation protocol (Eqs. 2-6) and its empirical validation against human ratings (Sec. 3.3, Table 5). The metric is defined independently of the human labels: Sglobal and S_rep^(k) are fixed cosine-similarity functions of DINOv3 embeddings and SAM3 masks, and the reported Sbar is an aggregate of those functions. No parameter of the metric is fitted to the 76-participant ratings; the human study is an external anchor collected after the protocol was fixed. The use of DINOv3 in dataset construction (Sec. 2.3) and in the metric (Eqs. 3, 6) shares a feature extractor, but this does not make the human-correlation result true by construction—the reported correlations could in principle have been low. The only self-citation is the Qwen-Image technical report used to identify the Qwen-Editor baseline; it is descriptive and not load-bearing. The Table 5 vs. Table 7 discrepancy over the definition and aggregation of Sbar is an internal reporting/consistency issue, not a circular reduction: nothing in the paper's equations forces the claimed rho_s=0.933 or rho_k=0.833 values. The paper's own limitation statement about foundation-model biases concerns data quality and annotation risk, not circularity. I therefore find no step that reduces a prediction to its inputs by definition.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

OpenVTON-Bench's central claims rest on hand-set parameters, foundation-model reliability, and unverified data-construction correctness; none of these are machine-checked or externally benchmarked in the paper.

free parameters (5)
  • Number of semantic clusters = 20
    Hand-chosen cluster count (Sec. 2.3) defines the '20 fine-grained categories' that underpin the balanced-sampling claim; no ablation shows sensitivity to this number.
  • Resolution selection bounds = 1024 <= min(H,W), max side <= 1536
    This filter turns 3M raw samples into ~300K candidates (Sec. 2.1) and defines the high-resolution positioning; the bounds are arbitrary commercial choices.
  • Erosion depth and structuring element = K=4 (k=0..3); B=3x3
    Multi-scale metric (Eq. 5, App. D.2) uses four erosion levels and a square 3x3 element; no sensitivity analysis, so the boundary-vs-texture separation claim depends on them.
  • Multi-scale aggregation rule = arithmetic mean of S_rep^(0..3)
    Final fidelity is the simple average (App. D.2); other aggregations could change method rankings and the reported correlations.
  • VLM final weighted score = weights unspecified
    Prompt asks for a 'final weighted score' without stating weights (App. B.1); if it is an unweighted average, that is a hidden design choice affecting correlation.
axioms (7)
  • domain assumption DINOv3 feature cosine similarity is monotonically related to human-perceived identity and texture fidelity (Eqs. 3, 6).
    The representation metric's validity rests on this unproven perceptual claim; no calibration against human ratings beyond aggregate correlations.
  • domain assumption SAM3 + GroundingDINO segment garments accurately in both ground-truth and generated images (Eq. 4, App. A.3).
    Incorrect masks would mix background into garment features and corrupt S_rep.
  • domain assumption Human annotators correctly enforced one-to-one garment-to-model matching for ~300K candidates (Sec. 2.2/A.1).
    If pair verification is wrong, ground-truth triplets are invalid.
  • domain assumption Qwen-VL-Plus produces unbiased five-dimensional scores aligned with human perception (Sec. 3.2.1/B.1).
    VLM-as-a-Judge is known to suffer prompt bias and hallucination; the paper relies on it as a core evaluation axis.
  • domain assumption 76-participant human ratings (>90K judgments) form an unbiased perceptual ground truth (Sec. 3.3/D.3).
    No inter-annotator agreement or demographic/sampling details are given.
  • ad hoc to paper The five VTON dimensions are orthogonal and sufficient (Sec. 3.2).
    The taxonomy is invented for this benchmark and is not derived from theory or user studies.
  • standard math Morphological erosion of paired masks separates boundary alignment from internal texture (Eq. 5).
    This geometric property is standard; the assumption is that DINO features of eroded regions track internal texture.

pith-pipeline@v1.3.0-alltime-deepseek · 19164 in / 24439 out tokens · 261518 ms · 2026-08-03T06:28:52.125841+00:00 · methodology

0 comments
read the original abstract

Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck. Traditional metrics struggle to quantify fine-grained texture details and semantic consistency, while existing datasets fail to meet commercial standards in scale and diversity. We present OpenVTON-Bench, a large-scale benchmark comprising approximately 100K high-resolution image pairs (up to $1536 \times 1536$). The dataset is constructed using DINOv3-based hierarchical clustering for semantically balanced sampling and Gemini-powered dense captioning, ensuring a uniform distribution across 20 fine-grained garment categories. To support reliable evaluation, we propose a multi-modal protocol that measures VTON quality along five interpretable dimensions: background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism. The protocol integrates VLM-based semantic reasoning with a novel Multi-Scale Representation Metric based on SAM3 segmentation and morphological erosion, enabling the separation of boundary alignment errors from internal texture artifacts. Experimental results show strong agreement with human judgments (Kendall's $\tau$ of 0.833 vs. 0.611 for SSIM), establishing a robust benchmark for VTON evaluation.

Figures

Figures reproduced from arXiv: 2601.22725 by Chenhui Wu, Jingwen Luo, Jin Li, Kai Wen, Shuai Jiang, Siqi Yin, Tao Chen, Weijie Wang.

Figure 1
Figure 1. Figure 1: The Proposed Hybrid Evaluation Framework. We move beyond single-scalar metrics by decomposing VTON quality into five human-aligned dimensions. Uniquely, our framework combines (Top) a VLM-as-a-Judge module for semantic auditing with (Bottom) a Multi-Scale Representation Metric that verifies semantic structural consistency. This synergy ensures both seman￾tic plausibility and accurate garment replication. e… view at source ↗
Figure 2
Figure 2. Figure 2: Data Construction Pipeline of OpenVTON-Bench. The process consists of three stages: (1) Large-scale raw data aggregation from diverse sources; (2) Hybrid annotation combining human verification for pair alignment and VLM-based dense captioning; (3) Semantic-aware filtering using DINOv3 clustering to ensure a balanced distribution across 20 fine-grained categories. 2. OpenVTON-Bench Constructing a benchmark… view at source ↗
Figure 3
Figure 3. Figure 3: Dataset Analysis of OpenVTON-Bench. Left (a-d): t-SNE visualizations of the full dataset and the train/validation/test splits. Right (e): Category distribution of the dataset. 2.4. Benchmark Overview and Statistics The final OpenVTON-Bench comprises 99,925 high￾resolution image pairs, establishing itself as one of the largest VTON benchmarks with consistent high fidelity. As summarized in [PITH_FULL_IMAGE… view at source ↗
Figure 4
Figure 4. Figure 4: Representative examples from OpenVTON-Bench. More examples are provided in Appendix D.4. 3.1. Preliminaries and Notation Let D = {(Ip, Ig, Igt)} denote the evaluation dataset, where Ip is the cloth-agnostic person image, Ig is the target gar￾ment image, and Igt is the ground-truth try-on result. A virtual try-on model G produces a synthesized image as defined in Eq. 1. ˆI = G(Ip, Ig), (1) The generated ima… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of state-of-the-art methods on OpenVTON-Bench. 3.3. Subjective Evaluation To validate the reliability of the proposed objective metrics, we conduct a large-scale human perceptual study. A total of 76 participants provided over 90,000 valid ratings by evalu￾ating randomly sampled result triplets (Ig, Igt, ˆI). In each triplet, ˆI represents the VTON result generated from the cor￾respo… view at source ↗
Figure 6
Figure 6. Figure 6: Correlation Analysis against Human Preference. The scatter plot compares objective metrics (normalized) with human ratings. Our Representation Metric (Red) and VLM Metric (Blue) show strong positive correlations. that for any given pair of models, our metric is the most likely to correctly predict which one a human would prefer, far exceeding standard metrics like SSIM (ρk = 0.611). 5. Related Works 5.1. V… view at source ↗
Figure 7
Figure 7. Figure 7: Diversity of OpenVTON-Bench. We display representative triplets (Garment, Model, Mask) sampled from the 20 distinct semantic clusters defined in Appendix D.4. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison of different models. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

    cs.CV 2026-07 accept novelty 7.0

    CtrlVTON recasts virtual try-on as mask-conditioned editing and introduces VIP-SAM for instance-level garment segmentation, beating proprietary editors on layout fidelity while matching garment quality.

  2. VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On

    cs.CV 2026-03 conditional novelty 6.0

    VTEdit-Bench and VTEdit-QA show top universal multi-reference editors match specialized VTON models on standard tasks and transfer more stably to harder multi-person/multi-cloth settings, yet still fail under complex ...

Reference graph

Works this paper leans on

6 extracted references · cited by 2 Pith papers

  1. [1]

    ”reasoning” object with 4 analysis fields (each max 100 words)

  2. [2]

    reasoning

    [Ground Truth Image]:The real photo of the person wearing the clothes (Reference for identity, pose, and background). 3.[Generated Image]:The AI-generated try-on result. Task:Compare the [Generated Image] against the references and score it on 5 specific dimensions. Use a scale of 1 to 5 (where 5 is perfect/identical, and 1 is failure). User Prompt:Please...

  3. [3]

    ”scores” object with all 5 numeric scores (1.0-5.0)

  4. [4]

    ”final weighted score” (1.0-5.0)

  5. [5]

    NO extra text outside the JSON

  6. [6]

    Representation-based Metrics We compute the Cosine Similarity between the [CLS] tokens of the Reference Garment and the Generated Image using DINOv3(Sim ´eoni et al., 2025)

    NO missing fields B.2. Representation-based Metrics We compute the Cosine Similarity between the [CLS] tokens of the Reference Garment and the Generated Image using DINOv3(Sim ´eoni et al., 2025). To isolate the garment region, we apply the generated segmentation mask to the try-on result before feature extraction. C. Experimental Details C.1. Hardware an...

    2025