{"id":"3187ec09-d65d-4fd9-b6fe-f511d9f7ac71","arxiv_id":"2603.13057","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"VTON-IQA is a reference-free, human-aligned quality scorer for virtual try-on, trained and validated on VTON-QBench (62,688 images, 431,800 annotations from 13,838 people).","lead":"The authors introduce VTON-IQA, a reference-free way to score virtual try-on images by how humans rate them, plus VTON-QBench, a large human-labeled set of 62,688 try-on images. Fashion e-commerce needs quality scores when no ground-truth photo of the same person in the target garment exists.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Generalizability of VTON-QBench annotations (from only 14 fixed generators) to unseen people, garments, and future VTON models is the softest support for the 'reliable human-aligned' claim.","rationale":"Only the abstract is present in the supplied source (FULL MANUSCRIPT TEXT block is empty), so no metrics, splits, ablations, or release statements can be inspected. The reader's weakest-assumption diagnosis is therefore exactly the load-bearing point: the annotations are treated as a universal human-quality oracle, yet they are generated exclusively by a closed set of 14 models. That assumption is necessary for the claim of reliable, general human alignment; nothing in the abstract secures it. Because the reader already correctly set UNVERDICTED / LOW confidence for the same reason, no verdict adjustment is warranted. A full-text pass that supplies leave-one-out numbers, inter-annotator agreement, and held-out-generator results could raise the verdict; until then the concern stands and the status remains UNVERDICTED.","tokens_in":2314,"tokens_out":617,"duration_ms":14801,"concrete_test":"If the full manuscript reports PLCC/SRCC, recompute them under a strict leave-one-generator-out protocol (train on 13 models, test on the held-out 14th, average over all 14 folds). If mean correlation falls more than ~0.15 relative to the in-distribution numbers claimed in the paper, the generalizability premise fails and the 'reliable' claim must be narrowed. Alternatively, annotate ~500 images from two VTON models never among the original 14 and measure VTON-IQA vs. human rank correlation; SRCC < 0.65 would confirm the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that VTON-IQA delivers reliable reference-free, human-aligned quality scores rests on VTON-QBench's 431k annotations (from 13,838 annotators on 62k images of 14 generators) constituting a stable, transferable target of human perceptual quality. Because every training/evaluation image is produced by one of those 14 models, the learned scorer can latch onto generator-specific artifacts rather than garment-person fidelity that humans care about. The abstract asserts 'extensive experiments show reliable' alignment, yet supplies no evidence of leave-one-generator-out, cross-person, or truly held-out-model correlations. If those splits are absent or weak, the model may appear human-aligned only inside the closed set of the 14 generators and fail to remain aligned on new architectures, body shapes, or garments—exactly the real-world regime the paper claims to serve. This is the single condition that must hold for the strongest claim to survive; everything else (scale of the benchmark, reference-free design) is secondary.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript proposes VTON-IQA, a reference-free image quality assessment framework for virtual try-on (VTON) that aims to predict human perceptual quality without ground-truth images of the same person wearing the target garment. To train and evaluate the approach, the authors introduce VTON-QBench, described as a large-scale human-annotated benchmark of 62,688 try-on images produced by 14 representative VTON models together with 431,800 quality annotations from 13,838 qualified annotators, claimed to be the largest subjective VTON evaluation set to date. The abstract asserts that extensive experiments demonstrate reliable human-aligned assessment and that the authors further use VTON-IQA to benchmark the 14 generators. The central claim is therefore that a model fit to this annotation pool recovers human judgments in the reference-free regime relevant to fashion e-commerce.","tokens_in":2565,"tokens_out":990,"duration_ms":14758,"significance":"If the empirical claims hold under rigorous held-out evaluation, the work would address a genuine practical gap: VTON systems are hard to score without paired ground-truth try-on images, and a stable, human-aligned reference-free metric would be useful for model selection and product deployment. The scale of VTON-QBench (tens of thousands of images and hundreds of thousands of annotations) is itself a potentially valuable community resource. Those contributions, however, depend entirely on whether the learned scorer generalizes beyond the closed set of 14 generators and the collected annotator pool, and on whether the reported human alignment is measured with independent splits, inter-annotator agreement, and leave-one-generator-out or truly unseen-model protocols. Without those results being verifiable, significance remains conditional.","major_comments":[{"comment":"The abstract asserts that “extensive experiments show that VTON-IQA achieves reliable human-aligned image quality assessment,” yet the provided manuscript body contains no correlation metrics (e.g., PLCC/SRCC/KRCC against human scores), no train/validation/test protocol, no inter-annotator agreement statistics, no ablations, and no error analysis. The central empirical claim is therefore unsupported by the text available for review and cannot be assessed for soundness.","section":null},{"comment":"All 62,688 images are generated by a fixed set of 14 VTON models. Without leave-one-generator-out, cross-person, cross-garment, or truly held-out-architecture evaluations, VTON-IQA may learn generator-specific artifacts rather than the garment–person fidelity humans care about. The abstract does not establish that the scorer remains human-aligned on people, garments, or future generators outside VTON-QBench—the regime the paper claims to serve. This is load-bearing for the “reliable reference-free” claim.","section":null},{"comment":"Human-feedback IQA is trained to match human labels; reporting alignment on the same annotation distribution is expected by construction. The manuscript must show that evaluation uses held-out annotators, images, and preferably models, and must report agreement among the 13,838 “qualified” annotators. Absent those details, the human-alignment claim risks being circular or overstated.","section":null}],"minor_comments":[{"comment":"The full manuscript body beyond the abstract was empty in the review package (only title, abstract, and metadata present). A complete technical review of methods, equations, tables, and figures is therefore impossible from the supplied source.","section":null},{"comment":"Clarify how “qualified annotators” were screened and how quality control (e.g., gold questions, consistency filters) was applied when collecting the 431,800 annotations.","section":null},{"comment":"State explicitly whether VTON-IQA is a learned regressor/ranker on top of a frozen backbone, an end-to-end model, or a prompt-based VLM, and what inputs it receives (person image, garment, try-on result only, etc.).","section":null}],"recommendation":"uncertain","confidential_remarks":"Only the abstract and empty body were available in the cacheable source; I cannot confirm whether the full PDF exists elsewhere. Recommendation is “uncertain” rather than reject because the claimed dataset scale and problem setting are potentially valuable if the missing protocols and metrics are solid. If the full paper is supplied, the single highest-priority check is leave-one-generator-out / held-out-model human correlation; if that is weak or absent, major_revision or reject would be appropriate. Scope fits cs.CV / multimedia evaluation venues."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a measurement paper for virtual try-on. They built VTON-QBench (62k try-on images from 14 models, ~432k ratings from ~14k annotators) and trained VTON-IQA, a reference-free scorer meant to match those human judgments, then used it to rank the same 14 systems.\n\nWhat is actually new is the scale of the human labels and the domain packaging, not a new IQA theory. Reference-free IQA and preference learning are established; applying them to VTON with a large, multi-model corpus is a solid engineering contribution for fashion e-commerce, where you almost never have a true ground-truth image of the same person in the target garment. Framing the problem that way is right, and a shared human-aligned score for model selection is something practitioners would use.\n\nSoft spots, in proportion: we only have the abstract. There are no SRCC/PLCC numbers, no inter-annotator agreement, no train/test protocol, no ablations, and no release statement. The stress-test concern is the real one: every image comes from one of 14 fixed generators, so a scorer can latch onto generator-specific artifacts instead of garment–person fidelity. Without leave-one-generator-out, cross-person, or truly held-out-model correlations, “reliable human-aligned” only means “matches annotators inside this closed set.” That is the load-bearing assumption for real-world use; everything else is secondary. Circularity is not fatal if held-out people/images are used—that is standard—but the abstract does not show it.\n\nWho it is for: fashion CV and e-commerce eval people, not general IQA theorists. Citation pattern and math cannot be judged from the abstract alone; the program itself is coherent.\n\nI would send it to peer review. Dataset scale alone deserves referee time if the full paper has proper splits, metrics, and artifact plans. I would not cite or rely on the rankings until those generalizability checks are visible.","headline":"Big human VTON quality bench plus a reference-free scorer—useful for fashion eval, but the “reliable alignment” claim is still uncheckable from the abstract and may not travel past the 14 training generators.","tokens_in":3218,"tokens_out":540,"would_cite":false,"duration_ms":12760,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Virtual try-on quality can be scored without a ground-truth photo by training on a large human preference set.","keywords":["virtual try-on","image quality assessment","reference-free IQA","human feedback","VTON-QBench","fashion e-commerce","perceptual quality"],"falsifier":"Hold out a fresh set of person-garment pairs and a previously unseen try-on generator, collect new human ratings under the same protocol, and check whether VTON-IQA's ranking and absolute scores still correlate strongly with those new ratings; a large drop would falsify the claimed human alignment.","tokens_in":3202,"feed_emoji":"👗","tokens_out":671,"duration_ms":4809,"temperature":0.7,"pith_summary":"Virtual try-on systems dress a person in a new garment from separate photos, but real deployments almost never have a true photo of that same person already wearing the garment, so ordinary image-quality scores that need a reference image do not apply. This paper builds a large human judgment set, VTON-QBench, of 62,688 try-on images from 14 generators labeled by 13,838 qualified annotators, then trains VTON-IQA, a reference-free scorer that predicts those human quality judgments. The claim is that the resulting scores track human preference well enough to rank models and diagnose failures without any ground-truth try-on photo. A sympathetic reader cares because fashion e-commerce needs automatic quality checks that work at scale on real customer photos; a human-aligned, reference-free metric is the missing piece that lets teams improve generators without constant expensive re-annotation.","feed_headline":"Try-on quality scored without a ground-truth photo","feed_subtitle":"A 62k-image human preference set trains a reference-free scorer that ranks 14 virtual try-on models.","key_machinery":"VTON-QBench: a large human preference corpus that supplies the training target for VTON-IQA, the reference-free scorer that maps a single try-on image to a predicted quality score without any ground-truth image of the same person in the garment.","core_discovery":"VTON-IQA is a reference-free image quality assessor for virtual try-on that aligns with human perceptual judgments. It is trained and validated on VTON-QBench, a new human-annotated corpus of 62,688 try-on images produced by 14 representative generators and 431,800 quality annotations from 13,838 qualified annotators, claimed as the largest such subjective set for VTON. Experiments show the scorer remains human-aligned, and the same scorer is used to rank the 14 generators.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Human feedback trains reference-free VTON quality scorer","62k-image VTON bench yields human-aligned quality scores","No ground truth: rank 14 try-on models via human prefs","VTON-IQA scores try-ons to match 431k human judgments","Reference-free scorer ranks VTON models by human taste"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The paper assumes that quality labels collected from its 13,838 annotators on images from 14 chosen generators form a stable, general target of human perceptual quality that will still match human judgment on new people, garments, and generators outside that set.","fun_headline_variants_meta":{"raw":{"variants":["Human feedback trains reference-free VTON quality scorer","62k-image VTON bench yields human-aligned quality scores","No ground truth: rank 14 try-on models via human prefs","VTON-IQA scores try-ons to match 431k human judgments","Reference-free scorer ranks VTON models by human taste"]},"model":"grok-4.5","effort":"low","cost_usd":0.004594,"raw_usage":{"total_tokens":1341,"prompt_tokens":767,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":45940000,"prompt_tokens_details":{"text_tokens":767,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":499,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":767,"tokens_out":75,"duration_ms":6388,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T21:53:04.996077+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Hold out a fresh set of person-garment pairs and a previously unseen try-on generator, collect new human ratings under the same protocol, and check whether VTON-IQA's ranking and absolute scores still correlate strongly with those new ratings; a large drop would falsify the claimed human alignment.","supporting_citations":[],"review_version":1}