{"id":"cca68901-8c24-4b26-9b3d-04b7b8de0298","arxiv_id":"2508.12605","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A UGC distortion instruction dataset with 11.5K images and a human-revised benchmark improves MLLM distortion grounding, perception, and quality description, with several tuned models passing GPT-4o on the benchmark.","lead":"This paper creates a large dataset and benchmark for teaching vision-language AI systems to explain photo quality by locating and describing distortions such as blur, noise, and compression. Fine-tuning several open models on this data improved their distortion grounding, perception, and quality descriptions, with some versions scoring above GPT-4o on the authors' benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ViDA-tuned models 'surpassing GPT-4o' is not yet supported: benchmark references, training targets, and the description judge all derive from GPT-4o, and human revision is asserted but unquantified, so the margin may reflect style mimicry rather than human-aligned quality analysis.","rationale":"The reader's weakest assumption is exactly the circularity I would flag: GPT-4o generates the attribute descriptions and CoT rationales used as training targets, and the same model is used to judge description quality against those GPT-4o-derived references. My reading of the supplementary material confirms this: Tables 3, 6, and 7 define GPT-4o as the judge, and Steps 2, 3, and 4 of the pipeline use GPT-4o to generate low-level attributes, descriptions, and QA pairs. Human revision of the benchmark is real mitigation, but it is not characterized quantitatively; no inter-annotator agreement, revision counts, or examples of substantial revisions are provided, so the extent to which residual GPT-4o output style remains is unknown. This matters most for the 'even surpassing GPT-4o' headline: on ViDA-UGC-Bench every ViDA-UGC-tuned model beats GPT-4o by a large margin, e.g., GPT-4o at 55.20% versus InternVL2.5-ViDA at 74.80% on perception, and a model trained on GPT-4o text should be expected to match GPT-4o's own judging preferences. The distortion boxes are human-annotated, and the Q-Bench perception results use human-created MCQs, so the dataset's value for grounding and general perception does not fall with this concern. I therefore would not reject the paper; the CONDITIONAL verdict remains appropriate, with the conditions being artifact release and an external human-judged evaluation of the headline claim.","tokens_in":25411,"tokens_out":5913,"duration_ms":60173,"concrete_test":"Run a blinded external evaluation on a random 100-image subset of ViDA-UGC-Bench: have the professional team plus one independent IQA rater group directly compare ViDA-UGC-tuned Qwen2-VL-7B and InternVL2.5-8B descriptions against GPT-4o descriptions on completeness, precision, and reasoning, without knowing which output comes from which system. Compute the human preference margin and compare it with the GPT-4o-judged margins in Table 2. If human preference is not significantly positive for the tuned models, or if GPT-4o wins, the 'even surpassing GPT-4o' claim is an artifact of using GPT-4o as both target generator and judge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two legs: (i) ViDA-UGC instruction tuning improves detailed IQA abilities, and (ii) the tuned models 'even surpass GPT-4o.' Leg (ii) rests on a closed GPT-4o loop. In Step 2 of the construction pipeline, 'GPT-4o then outputs textual descriptions of distortions and their visual attributes' from human boxes; in Step 3, 'GPT-4o produces a logical reasoning process' under the CoT framework; in Step 4, GPT-4o converts descriptions into VQA and MCQ. The ViDA-UGC-Bench is then selected from this same corpus, and its description evaluation uses 'a five-round GPT evaluation' (supplementary Tables 3, 6, 7) between the model output and '[DISTORTION INFO]', the same attribute representation GPT-4o produced in Step 2. A professional team is said to revise the benchmark QA pairs, but the paper reports no revision rate, no inter-annotator agreement, and no before/after comparison. Because a model fine-tuned on hundreds of thousands of GPT-4o-written rationales is well positioned to match GPT-4o's judging preferences, the 'surpassing GPT-4o' conclusion is exactly what style mimicry would produce. The concern is not that the dataset is useless; perception and grounding gains on human-annotated MCQ and box metrics are credible. The headline superiority claim, however, needs an external, non-GPT-4o evaluation before it can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ViDA-UGC, a large instruction-tuning dataset for explainable image quality assessment of user-generated content. The dataset contains 11,534 images, 36K human-annotated distortion bounding boxes, and 534K instruction samples spanning three sub-datasets: distortion grounding, low-level perception, and reasoning quality description. The construction pipeline combines human annotation with GPT-4o generation under a Chain-of-Thought assessment framework. The authors also introduce ViDA-UGC-Bench (476 images, 6,149 QA pairs) with professional human revision. Experiments fine-tune four open MLLMs on ViDA-UGC and compare against Q-Instruct tuning and GPT-4o zero-shot on Q-Bench and ViDA-UGC-Bench, reporting consistent improvements across perception, description, and grounding.","tokens_in":25667,"tokens_out":9258,"duration_ms":83767,"significance":"If validated, the dataset and benchmark are potentially valuable contributions: they are among the first large-scale UGC-specific distortion assessment corpora covering three complementary tasks, with human-in-the-loop construction, a distortion-oriented CoT framework that also works training-free, and an external anchor via Q-Bench. The grounding results are particularly credible because the bounding boxes are human-annotated and the gains are large (e.g., Table 3, Acc0.5 improvements of +8.9 to +18.4 over baselines). The paper also includes useful comparisons with object-detection baselines. However, the headline claim that ViDA-UGC-tuned models 'even surpass GPT-4o' is weakened by GPT-4o self-reference in the benchmark construction and evaluation; this needs an external, human-grounded evaluation before the claim can be accepted.","major_comments":[{"comment":"The claim that ViDA-UGC-tuned models surpass GPT-4o is not yet supported because the benchmark references, training targets, and description judge all derive from GPT-4o. In Step 2 of the construction pipeline, GPT-4o generates the distortion attributes (type, position, severity, impact, significance); in Step 3, GPT-4o produces the CoT reasoning descriptions; in Step 4, GPT-4o converts them into MCQ/VQA. ViDA-UGC-Bench is selected from this corpus, and the description evaluation in the supplementary (Tables 3, 6, 7) uses GPT-4o as the judge, scoring model outputs against [DISTORTION INFO], the same attribute representation GPT-4o produced in Step 2. A model fine-tuned on hundreds of thousands of GPT-4o-written rationales is therefore well positioned to match GPT-4o's output style and score highly on its judging preferences; the reported margins on ViDA-UGC-Bench description (e.g., InternVL3-8B-ViDA 5.87 vs baseline 4.84 in Table 2) may reflect style mimicry rather than human-aligned quality analysis. The human expert revision is asserted but unquantified: no revision rate, inter-annotator agreement, or before/after comparison is reported. Please (i) evaluate quality descriptions with human expert judgments or with a different judge model and human-written references, (ii) report quantitative revision statistics for ViDA-UGC-Bench, and (iii) clearly separate the externally anchored Q-Bench perception results from the in-corpus ViDA-UGC-Bench results when making 'surpassing GPT-4o' statements.","section":"Proposed ViDA-UGC-Bench; Supplementary Benchmark on Quality Description Ability"},{"comment":"The disjointness of ViDA-UGC-Bench from the training data is not established. The main text says the benchmark is composed of samples 'from ViDA-UGC', and the models are fine-tuned on ViDA-UGC; the supplementary only says 'we split the whole dataset into train and benchmark' without giving the number of training images or the exact split procedure. If any of the 476 benchmark images or their 6,149 QA pairs were also in the instruction-tuning corpus, the gains in Tables 1-3 would reflect memorization rather than generalization. Please state the split explicitly, report the number of training images, and confirm that no benchmark image, MCQ, grounding sample, or description reference appears in the fine-tuning data.","section":"Proposed ViDA-UGC-Bench; Supplementary Data Statistics"},{"comment":"The experimental comparisons lack statistical support for the headline margin. Table 1 shows Qwen2-VL-7B-ViDA at 80.6% vs GPT-4o at 78.6% on Q-Bench overall; with no confidence intervals, significance tests, or multiple-seed runs, a 2-point margin on a several-thousand-question benchmark is not established as a real difference. Similarly, Table 3 reports Acc0.5 gains up to +18.4 without error bars or significance testing. Please report bootstrap confidence intervals or paired significance tests (e.g., McNemar's test) for the main perception and grounding comparisons, and adjust the abstract's 'even surpassing GPT-4o' claim to the actually supported comparisons.","section":"Experiment, Table 1 and Table 3"}],"minor_comments":[{"comment":"The entry '77.19/%' for InternVL3-8B ViDA-UGC should read '77.19%'.","section":"Table 1"},{"comment":"The typo 'Q-Insturct' appears in the description of Table 2; it should be 'Q-Instruct'.","section":"Main Results, Description paragraph"},{"comment":"No GPT-4o row is reported in the description results (Table 2), so the abstract's broad statement that tuned models 'even surpass GPT-4o' is not directly supported for the description task; please either add GPT-4o description scores or restrict the superiority claim to the perception task.","section":"Abstract and Table 2"},{"comment":"The sentence 'we are the first to transfer the task from high-level object grounding to low-level distortion grounding' appears to conflict with the cited prior works Q-Ground and Grounding-IQA, which already perform distortion grounding; please rephrase to specify the precise novelty (e.g., the referring-grounding formulation for distortions).","section":"Proposed ViDA-UGC Dataset, Step 2"},{"comment":"The dataset size is referred to as both 534K instruction samples and 'ViDA-587K' in the supplementary; please make the numbers consistent and clarify the exact count.","section":"Supplementary, Dataset Statistics and Experimental Details"},{"comment":"For the Q-Bench rows of Table 2, it is unclear whether the golden reference is the human-written LLDescribe text or ViDA-UGC's [DISTORTION INFO]; the prompts in Tables 3/6/7 suggest the latter, which would make the Q-Bench description scores also dependent on GPT-4o-generated references. Please clarify the reference used for each benchmark.","section":"Supplementary, Benchmark on Quality Description Ability"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the main obstacle to acceptance. If the authors provide external human evaluation of descriptions, quantitative revision statistics, and explicit confirmation of train/benchmark disjointness, the paper could become acceptable. The dataset itself is potentially valuable to the community, and the grounding results appear credible because they rest on human-annotated boxes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jeremy,\n\nThe dataset is the real contribution; the headline model claim is only half-supported. What's new: a UGC-specific distortion instruction corpus, 11.5K images with human boxes and MOS, split into grounding, perception, and description tasks, and the first transfer of referring grounding to low-level distortion grounding. The ten-distortion taxonomy and the CoT construction pipeline are thoughtful, and the annotator qualification procedure is more careful than most work in this space.\n\nThe experiments are also more honest than typical: four base MLLMs, two benchmarks, consistent gains after ViDA-UGC tuning. The Q-Bench results matter because Q-Bench is external to this data pipeline — Qwen2-VL-7B tuned on ViDA-UGC reaches 80.6% vs GPT-4o's 78.6% overall. That is not circular, and it is the strongest evidence in the paper.\n\nThe soft spot is the description leg, and the stress-test note is right about that. GPT-4o generates the distortion triplets and CoT rationales that become both training targets and benchmark references, and GPT-4o also judges the model descriptions. The professional revision team is asserted but we get no revision rate, no inter-annotator agreement, no before/after comparison. So the description numbers, including the claim of beating GPT-4o there, could reflect style mimicry rather than human-aligned quality analysis. The perception and grounding legs are much cleaner because boxes and MOS are human and the MCQ answers are human-revised.\n\nMinor but real: 476 benchmark images is small; no confidence intervals or significance tests anywhere; the response-rate mismatch (0.72 vs 0.79) and the '77.19/%' typo are sloppy; and no dataset, code, or benchmark is released, which limits immediate use.\n\nWho gets value: researchers in explainable IQA and MLLM low-level vision. The paper deserves a serious referee. If the authors ship the artifacts, add an external human-judgment evaluation for descriptions, report revision statistics, and scope the 'surpassing GPT-4o' claim to Q-Bench perception, this becomes a genuinely reusable resource.\n\nRecommendation: send to peer review, with those requirements.","headline":"A genuinely useful UGC distortion dataset and benchmark with credible gains on external Q-Bench, but the 'surpassing GPT-4o' claim overreaches: the description evaluation is a closed GPT-4o loop.","tokens_in":26290,"tokens_out":3088,"would_cite":true,"duration_ms":31527,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 534K-sample, distortion-focused instruction dataset gives open multimodal language models detailed image quality analysis for user-generated images — distortion grounding, low-level perception, and reasoning…","keywords":["image quality assessment","multimodal large language models","user-generated content","distortion grounding","explainable IQA","instruction tuning","chain-of-thought prompting","ViDA-UGC-Bench"],"falsifier":"Give independent human experts the same UGC images without the reference distortion triplets and ask them to mark each model-produced description for factual accuracy about distortion type, location, severity, and reasoning validity; if human ratings do not rank ViDA-UGC-tuned models above their base models, or if they disagree with the GPT-4o judge on the same texts, the reported improvements are evaluation artifacts rather than real quality-analysis gains.","tokens_in":25161,"feed_emoji":"🖼️","tokens_out":7809,"duration_ms":72171,"temperature":0.7,"pith_summary":"The paper aims to show that open multimodal language models can be taught to analyze image quality the way a human expert would: by locating distortions, naming their attributes, and reasoning from those observations to a quality judgment. The authors build ViDA-UGC, a dataset of 11,534 user-generated images with 36K human-labeled distortion boxes and 534K instruction-tuning samples, plus ViDA-UGC-Bench, a 476-image benchmark whose 6,149 questions were revised by a professional team. They pair the data with a chain-of-thought assessment framework that generates stepwise quality descriptions from human annotations and GPT-4o distortion descriptions. If the reported results hold, fine-tuning on this corpus gives open models distortion grounding, low-level perception, and reasoning description abilities that match or exceed GPT-4o, which is what practical quality control and restoration guidance require.","feed_headline":"Distortion-tuned MLLMs beat GPT-4o on image quality analysis","feed_subtitle":"A 534K distortion dataset lifts open MLLMs to expert-level grounding, perception, and reasoning about UGC flaws.","key_machinery":"The load-bearing mechanism is a four-step distortion-oriented construction pipeline. Images are sampled from UGC datasets with a MILP-based balancing strategy; human subjects draw distortion bounding boxes and give mean opinion scores. Each box is marked visually using a set-of-mark scheme, and GPT-4o is prompted to write five attributes for it: type, position, severity, impact, and significance. A chain-of-thought assessment framework then feeds the human MOS, rating criteria, and these distortion triplets into GPT-4o and asks it to analyze overall quality in five ordered steps — general impression, distortion search and analysis, key-distortion identification, overall quality analysis, and rating — and to assess individual distortions similarly, interleaving grounding as '[distortion](bounding box)' so text and location stay tied. Step 4 converts all of this into templated grounding, perception, and description instruction data. The framework does double duty: it generates the training corpus and, used as a prompt at inference time, improves description quality for models that were never fine-tuned.","core_discovery":"The paper's central claim is that distortion-specific data, not more general low-level data, is what unlocks detailed explainable IQA in open MLLMs. Concretely, on ViDA-UGC-Bench, tuning with ViDA-UGC lifts overall low-level perception accuracy from 47.53% to 71.45% for Qwen2-VL-7B and from 47.37% to 73.00% for InternVL3-8B, both surpassing GPT-4o's zero-shot 55.20%; tuning with Q-Instruct, by contrast, often lowers strong baselines' scores. Referring-grounding Acc 0.5 rises from about 24.9% to 42.1% for Qwen2-VL-7B and from 25.8% to 44.2% for InternVL3-8B. Quality-description scores also improve across all baselines, and the training-free CoT framework alone improves descriptions without any fine-tuning. The paper interprets these results as evidence that MLLMs have latent detailed-quality-analysis abilities that a distortion-oriented pipeline can reliably unlock.","pith_inferences":["The five-attribute distortion schema is content-agnostic; a testable extension is to annotate AIGC images with the same schema and see whether the grounding and description gains transfer, since the paper explicitly restricts itself to UGC.","The paper's tables show ViDA-UGC tuning can hurt performance on non-distortion low-level questions about color, lighting, and composition, so a mixed curriculum that combines ViDA-UGC with general low-level perception data may be needed for a model strong on both; this combination is not tested in the paper.","Because the benchmark's grounding labels are boxes rather than masks, the gap the paper reports between MLLMs and detection models could shrink or grow if the task were redefined as pixel-level region perception.","The pipeline could become a data flywheel: models tuned on ViDA-UGC could propose candidate distortion boxes for human review, expanding annotation to larger UGC streams at lower cost."],"forward_implications":["ViDA-UGC-tuned open models outperform GPT-4o zero-shot on detailed low-level perception of UGC distortions: the best tuned model scores 74.80% overall on ViDA-UGC-Bench versus GPT-4o's 55.20%.","Distortion referring-grounding accuracy, measured by Acc 0.5, rises by roughly 14 to 18 percentage points across four base models after tuning, and models that sometimes failed to return boxes become reliable.","Training with Q-Instruct degrades several strong baselines on ViDA-UGC-Bench, while ViDA-UGC improves them, implying that generic low-level instruction data is not a substitute for distortion-specific supervision.","Applying the CoT framework at inference time improves quality-description scores on both benchmarks even without fine-tuning, so the same prompting recipe can be used directly with existing MLLMs.","Because the benchmark shows an average 29% drop from Q-Bench to ViDA-UGC-Bench, prior results likely overstate MLLMs' readiness for detailed quality analysis."],"supporting_citations":[{"why":"Q-Bench supplies the general low-level perception and description evaluation that ViDA-UGC-Bench extends, and its GPT-based description scoring prompts are adapted.","marker":"Wu et al. 2024a"},{"why":"Q-Instruct is the main instruction-tuning baseline that ViDA-UGC is compared against, and its Q-Pathway descriptions are used for leakage-safe description evaluation.","marker":"Wu et al. 2024b"},{"why":"Q-Ground provides the prior distortion grounding dataset whose masks are compared with ViDA-UGC's boxes.","marker":"Chen et al. 2024a"},{"why":"DepictQA-Wild is the prior descriptive IQA dataset used for length and task-coverage comparisons.","marker":"You et al. 2024a"},{"why":"VisualCritic is the prior low-level instruction-tuning dataset that the paper situates itself against.","marker":"Huang et al. 2024"},{"why":"EvalMuse-40K supplies the MILP sampling strategy that balances image selection across low-level features.","marker":"Han et al. 2024"},{"why":"Set-of-Mark provides the visual marking method used to show GPT-4o which regions correspond to each distortion.","marker":"Yang et al. 2023"},{"why":"Chain-of-thought prompting is the basis for the step-wise assessment framework that generates the description data.","marker":"Wei et al. 2022"}],"fun_headline_variants":["Distortion tuning beats GPT-4o on explainable image quality","Open MLLMs surpass GPT-4o with distortion-specific data","Why distortion data unlocks expert-level quality analysis","UGC quality: open models outpace GPT-4o after distortion tuning","Distortion-first training lifts open MLLMs past GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument stands on treating GPT-4o's distortion descriptions, generated from human boxes and scores and then partly edited by a professional team, as the true explanation of image quality — if those descriptions carry systematic biases that the human edits did not remove, models tuned and scored against them may be learning to mimic a particular judge rather than to analyze quality.","fun_headline_variants_meta":{"raw":{"variants":["Distortion tuning beats GPT-4o on explainable image quality","Open MLLMs surpass GPT-4o with distortion-specific data","Why distortion data unlocks expert-level quality analysis","UGC quality: open models outpace GPT-4o after distortion tuning","Distortion-first training lifts open MLLMs past GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000729,"raw_usage":{"total_tokens":3332,"prompt_tokens":1084,"completion_tokens":2248,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2159}},"tokens_in":700,"tokens_out":2248,"duration_ms":16241,"temperature":1.0,"reasoning_tokens":2159,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:21:17.561659+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give independent human experts the same UGC images without the reference distortion triplets and ask them to mark each model-produced description for factual accuracy about distortion type, location, severity, and reasoning validity; if human ratings do not rank ViDA-UGC-tuned models above their base models, or if they disagree with the GPT-4o judge on the same texts, the reported improvements are evaluation artifacts rather than real quality-analysis gains.","supporting_citations":[],"review_version":2}