REVIEW 4 major objections 4 minor 1 references
Better Supervised Fine-tuning for VQA: Integer-Only Loss
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that a fine-tuning method called IOVQA can improve video quality assessment by rounding decimal MOS scores to integers and computing loss only on the first two digits of each label.
desk verdict A plausible little VQA fine-tuning trick buried in an unreadable PDF; no numbers, no ablations, can't verify anything yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the target-masked integer-only loss: labels are integer-rounded MOS scores in the range [10, 50], and the loss selectively unmask keeps only the first two digits of each label while masking the rest. This forces the gradient to concentrate on the leading digits of the score, while the range constraint keeps outputs numerically stable.
What would settle it
Fine-tune the same base model on the same dataset with the same schedule, but replace integer labels and the two-digit mask with original decimal MOS labels and full-label loss; if this control meets or beats the integer-mask variant on a held-out video quality benchmark, the central claim is falsified.
Extended reading notes
Core claim
IOVQA (Integer-only VQA) is a supervised fine-tuning recipe for video quality assessment. During dataset curation, Overall_MOS values are converted from decimals to integers, and the model's answer space is constrained to integers in [10, 50] to avoid numerical instability. At training time a target-mask is applied so the loss sees only the first two digits of the integer label; the rest are masked out. The paper reports that this focuses learning on the critical components of the numerical evaluation. Applied to Qwen2.5-VL, the method improves accuracy and consistency in video quality assessment and achieved 3rd place in Track I of the VQualA 2025 GenAI-Bench AIGC Video Quality Assessment C
Load-bearing premise
The claim collapses if the improvement seen in the challenge comes from dataset composition, training schedule, or the base model rather than from the integer labels and the two-digit target mask, or if rounding decimal MOS scores to integers discards information the model actually needs.
Editorial extensions
If this is right
- If integer-rounded labels suffice, future video quality datasets need not preserve decimal MOS precision for fine-tuning.
- Masking all but the first two digits implies the model learns the coarse score band before fine detail, which may simplify loss design for other quantitative evaluation tasks.
- Constraining output to [10, 50] avoids the numerical instability of unbounded regression, so the recipe may transfer to other vision-language models beyond Qwen2.5-VL.
- The third-place challenge result is a field-level demonstration that label construction alone can move performance in AIGC video quality assessment.
Reading between the lines
- Editorial inference: rounding and masking act as a form of output regularization, and it would be worth testing whether the benefit persists when the same data and training schedule are kept and only the masking pattern changes.
- Editorial inference: the same target-mask idea could be applied to other continuous regression outputs of vision-language models, such as aesthetic scores or relevance ratings, where leading digits carry most of the decision-relevant signal.
- Editorial inference: a direct ablation varying mask width (one digit, two digits, full integer, decimal) would separate how much of the gain comes from masking versus integer rounding; the paper's reported comparison does not by itself isolate these two factors.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes IOVQA, a fine-tuning method for vision-language models in video quality assessment. The claimed innovations are (i) converting decimal MOS labels to integers in the range [10,50] and constraining the model output to that integer range, and (ii) a target-mask loss that is said to unmask only the first two-digit integer of the label, forcing the model to focus on critical parts of the numerical evaluation. The method is applied to Qwen2.5-VL and the paper reports that it achieves 3rd place in the VQualA 2025 GenAI-Bench AIGC Video Quality Assessment Challenge Track I. The abstract asserts significant improvement in accuracy and consistency, but provides no quantitative results, no baseline comparison, no ablations, and no statistical analysis. The supplied full text is heavily corrupted, so the loss definition and experimental tables cannot be verified.
Significance. If the central claim were substantiated, the paper would offer a simple, computationally cheap fine-tuning trick for scalar-output VQA—integer-only labels and a masked loss—with potential value for video quality and aesthetic assessment. The use of an external challenge leaderboard is a strength insofar as it provides an independent evaluation setting, and the proposed design is concrete and falsifiable. However, the significance is currently unestablished: the evidence in the manuscript is limited to a single leaderboard rank, which is confounded by dataset, schedule, and base-model choices, and the target-mask operation appears possibly vacuous given the integer label range. The paper therefore needs substantial additional evidence before its claims can be assessed.
major comments (4)
- [Abstract and §4 (Experimental results)] The central claim that the method 'significantly improves' accuracy and consistency is not supported by any reported numerical result. The only evidence cited is a 3rd-place rank in an external challenge, with no comparison to a baseline fine-tune of Qwen2.5-VL, no metric values, no error bars, and no statistical tests. A single leaderboard position is confounded by dataset composition, training schedule, hyperparameters, and base-model initialization, so it cannot isolate the contribution of the proposed components. A direct controlled comparison (same data and schedule, with and without the integer-label and target-mask design) is required.
- [§3, target-mask strategy] The masking rule is under-specified and possibly a no-op. Labels are converted to integers in [10,50]; every integer in that range has exactly two decimal digits. 'Only the first two-digit-integer of the label is unmasked' therefore either leaves the entire label unmasked or requires a token-level mask definition that is not given in the legible portions of the text. Please state precisely which tokens/positions are masked in the loss and provide an ablation that isolates the mask from the integer-label construction; otherwise the claimed novelty of the target mask cannot be evaluated.
- [§3, label construction and loss] The choice of the integer range [10,50] and the two-digit mask are free parameters. No sensitivity analysis or ablation is provided to justify these choices or to show they are responsible for any improvement. In particular, the paper does not compare integer labels with decimal labels, nor does it compare masked and unmasked losses. Without such evidence, the observed ranking could be driven by unrelated fine-tuning details rather than by the proposed design.
- [Full text, equations and results] The supplied manuscript is heavily corrupted: many passages are mojibake, and the running header cites arXiv:2508.11176v1 rather than arXiv:2508.11170. The loss equations, dataset description, training details, and result tables are not legible enough to be checked. This makes verification impossible in the current form. The authors should provide a clean, complete version before the technical content can be reviewed.
minor comments (4)
- [Header] The running header of the supplied PDF/plain text cites a different arXiv identifier (2508.11176v1) than the paper under review (2508.11170). This should be corrected.
- [Abstract] The abstract claims the contribution is 'merely leaving integer labels during fine-tuning,' but the method also introduces a target-mask loss; the phrasing is inconsistent and should be clarified.
- [Notation] The notation around 'Overall_MOS', 'MOS', and 'two-digit-integer' is inconsistent. Please define all symbols and use a consistent hyphenation and capitalization.
- [References] Provide a citable reference or URL for the VQualA 2025 challenge leaderboard, including the date of access, so the claimed 3rd-place result can be independently verified.
Circularity Check
No circularity found: the headline result is an external challenge ranking, and the proposed label/mask design is an input to training rather than a quantity derived from the evaluation it predicts.
full rationale
The claimed contribution is empirical: after fine-tuning Qwen2.5-VL with integer labels and a target-mask loss, the model ranked 3rd in VQualA 2025 GenAI-Bench AIGC Video Quality Assessment Challenge Track I (abstract). That result is measured against an external benchmark, not against a quantity constructed from the paper's own labels or loss. I find no visible equation in which the target-mask loss is defined to equal the challenge metric, and no parameter that is fit to a subset of the benchmark and then 'predicted' on a closely related subset. The integer range [10,50] and the rounding of decimal MOS are data-preparation choices, i.e., inputs to the fine-tuning; the paper does not define 'accuracy' as identical to the masked loss. No load-bearing self-citation or imported uniqueness theorem appears in the readable text. The full text is heavily corrupted, so the loss equations and experimental tables cannot be audited; the running header even cites arXiv:2508.11176v1 instead of 2508.11170. These are serious verification problems, and the abstract's causal attribution ('the proposed method significantly improves') lacks a visible ablation against a plain Qwen2.5-VL fine-tune. There is also an ambiguity in the target-mask description: because labels are already integers in [10,50], masking 'all but the first two-digit-integer' may be a no-op, which would make the mechanism vacuous rather than circular. Missing support and unverifiability are correctness/evidence concerns, not circularity. Under the hard rule that circularity requires a quotable by-construction equivalence, the appropriate finding is no significant circularity (0).
Assumptions & free parameters
free parameters (2)
- Integer output range [10,50]
- Target mask length (first two digits)
assumptions (2)
- domain assumption Integer conversion of decimal MOS labels preserves the information needed for VQA training
- domain assumption Masking all but the first two digits focuses learning on critical components
Cite this review
Pith. "Pith review of Better Supervised Fine-tuning for VQA: Integer-Only Loss." pith.science (2026). https://pith.science/paper/MTNBGGJQ
@misc{pith2026250811170,
author = {Pith},
title = {Pith review of: Better Supervised Fine-tuning for VQA: Integer-Only Loss},
year = {2026},
howpublished = {\url{https://pith.science/paper/MTNBGGJQ}},
note = {Machine review of arXiv:2508.11170}
}
read the original abstract
With the rapid advancement of vision language models(VLM), their ability to assess visual content based on specific criteria and dimensions has become increasingly critical for applications such as video-theme consistency assessment and visual quality scoring. However, existing methods often suffer from imprecise results and inefficient loss calculation, which limit the focus of the model on key evaluation indicators. To address this, we propose IOVQA(Integer-only VQA), a novel fine-tuning approach tailored for VLMs to enhance their performance in video quality assessment tasks. The key innovation of IOVQA lies in its label construction and its targeted loss calculation mechanism. Specifically, during dataset curation, we constrain the model's output to integers within the range of [10,50], ensuring numerical stability, and convert decimal Overall_MOS to integer before using them as labels. We also introduce a target-mask strategy: when computing the loss, only the first two-digit-integer of the label is unmasked, forcing the model to learn the critical components of the numerical evaluation. After fine-tuning the Qwen2.5-VL model using the constructed dataset, experimental results demonstrate that the proposed method significantly improves the model's accuracy and consistency in the VQA task, ranking 3rd in VQualA 2025 GenAI-Bench AIGC Video Quality Assessment Challenge -- Track I. Our work highlights the effectiveness of merely leaving integer labels during fine-tuning, providing an effective idea for optimizing VLMs in quantitative evaluation scenarios.
Reference graph
Works this paper leans on
-
[1]
������� �� � ���� ����� ������ ���� ��� ��� �� ������ ���� � ������������ ��� ����������� ��� ������ ������������ ������� �������� ������ ����� �� ������� ���� ����� ���� ����� ��� ����� ��� ��� ���������������������� ���������� ���� �������� ��������� ��� ���������� ����������� ��������������� ������ ������ �� �������� ������������� ������ ����� ������� ...
arXiv 2025
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.