Pith. sign in

REVIEW 4 major objections 6 minor 79 references

Scaling-up Perceptual Video Quality Assessment

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scaling video-quality instruction data to 400K pairs, built by a human-in-the-loop machine pipeline, yields state-of-the-art LMM video quality understanding and rating.

desk verdict Large, genuinely useful VQA instruction set with competitive rating results, but the fine-grained benchmark headline is inflated by an in-distribution machine half; the human half shows only a 0.51-point lead. read the letter →

arxiv 2505.22543 v1 pith:QF7XK2PE submitted 2025-05-28 cs.CV cs.AI

Ziheng Jia , Zicheng Zhang , Zeyu Zhang , Yingji Liang , Xiaorong Zhu , Chunyi Li , Jinliang Han , Haoning Wu
show 7 more authors
Bin Wang Haoran Zhang Guanyu Zhu Qiyong Zhao Xiaohong Liu Guangtao Zhai Xiongkuo Min
This is my paper
classification cs.CVcs.AI
keywords videoqualityassessmentlargemultimodalmodelsinstructiondatabasedatascalinglawunderstandingratinghuman-in-the-loopannotationspatiotemporal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that the data scaling law that has lifted large multimodal models on other tasks can be made to work for perceptual video quality assessment, if the bottleneck of expensive human labeling is circumvented. Its proposed solution, OmniVQA, builds a 400K-pair video-quality instruction database through a human-in-the-loop pipeline that is driven mostly by machine annotation, plus a 20K-video human-scored rating set. Trained on these with a task-specific complementary strategy, a 7B model (Chatter) reports state-of-the-art results on general video quality understanding, and its companion (Rater) reports best or second-best correlation on six public rating datasets. If that holds, large-scale quality-understanding data no longer requires expert-level human annotation at industrial cost.

What carries the argument

The mechanism that carries the argument is the rejection-sampling annotation chain combined with complementary training. In the technical branch, a domain-expert LMM is asked the same quality question five times with different phrasings; a reasoning LLM merges answers that recur at least three times into a summary and discards contradicted ones, a state-of-the-art LMM then votes 2/1/0 on whether the summary matches the keyframes, and a human expert resolves any zero-score vote. This converts direct annotation, which general LMMs do poorly, into a voting and verification task, which they do better. Complementary training then trains the same 7B base model sequentially on one task family before the other, letting the rating data act as pretraining for the understanding data instead of mixing both into the same batch. The in-context branch contributes a second mechanism: synthetic distortions of known location, duration, and severity, so the training signal for spatiotemporal questions has ground-truth geometry.

What would settle it

Take a held-out set of about 200 UGC videos with known human MOS, run the full technical-branch pipeline but corrupt the expert sampling by flipping the polarity of the five answers on half the videos, then fine-tune Chatter identically; if the corrupted version matches the reported Q-bench-video and OmniVQA-FG-Bench scores, the quality of the expert annotations is not carrying the result, while a collapse would show it is load-bearing. A complementary test is to evaluate Chatter on a fine-grained spatiotemporal quality benchmark built independently of the authors' training and evaluation data, where a large drop relative to same-group numbers would show that the understanding ability does not transfer.

Watch

Extended reading notes

Core claim

The central claim is that scaling, not architecture, is what is missing in perceptual video quality assessment, and that scaling can be done cheaply and reliably. The authors construct a machine-annotation pipeline organized into three branches: a technical branch that samples eight quality factors from a domain-expert model (VQA 2-Assistant), summarizes five paraphrasings with a reasoning LLM, and lets a stronger LMM vote on each summary with human judges called in when a vote scores zero; an in-context branch that synthesizes local spatial and temporal distortions with bounding boxes and semantic object descriptions so the model learns spatiotemporal localization; and an aesthetic branch that annotates style, spatiotemporal composition, and evoked emotion. They report that Chatter (7B) reaches 58.50% overall accuracy on Q-bench-video test and 59.08% on dev, and 65.32% overall on their new OmniVQA-FG-Benchmark versus 54.12% for VQA 2-Assistant, while Rater reaches 0.902 SRCC and 0.905 PLCC on LSVQ-test and top-two or top-three performance on five other rating sets. The paper's conclusion is that mostly machine-generated data, with humans only at the decision points, can substitute for fully manual annotation at scale.

Load-bearing premise

Everything rests on VQA 2-Assistant's single-factor annotations being trustworthy enough that a five-sample majority, an LLM summary, and an LMM vote can correct the occasional error; if that expert has systematic perceptual blind spots, the 400K set inherits them and Chatter's benchmark gains will not transfer to new videos.

Editorial extensions

If this is right

  • A mostly machine-generated, human-vetted instruction database of 400K pairs is enough to push a 7B model past general LMMs, including 72B ones, on video quality understanding benchmarks.
  • The data-scaling curve on Q-bench-video rises from 100K to 400K pairs but flattens at the 7B scale, so larger parameter counts, not just more data, are the next lever.
  • Training for quality rating first and quality understanding second, or vice versa, avoids the interference that random mixing causes and improves both tasks over direct training.
  • The in-context branch produces a large margin in spatiotemporal fine-grained understanding: Chatter scores 69.87% on the machine-annotated fine-grain subset versus 55.33% for VQA 2-Assistant.
  • A human-labeled 20K MOS dataset with hidden-reference supervision gives the best or second-best rating correlation across LSVQ, KoNViD-1k, LIVE-VQC, YT-UGC, and the new MOS-20K test.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same three-branch pipeline could be pointed at image quality assessment or other perceptual judgment tasks; the paper describes the cost falling to near zero once open-source LMMs replace paid APIs, but does not test that regime.
  • The expert model can be iteratively replaced: after Chatter is trained, it could re-annotate the technical branch and bootstrap a second-generation 400K instruction database, a recursion the paper mentions as a possible application but does not run.
  • The reported understanding gains are measured on benchmarks drawn from the same data ecosystem as the training set; on an independent fine-grained benchmark built by a different group, the margin over VQA 2-Assistant could shrink if the bias of the expert judge chain transfers.
  • The in-context branch's synthetic distortions, with known time intervals and bounding boxes, could serve as a data engine for video grounding and temporal anomaly localization beyond quality assessment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes OmniVQA, a framework for scaling perceptual video quality assessment instruction data through a human-in-the-loop, machine-annotation pipeline. It introduces OmniVQA-Chat-400K, a 400K-pair MIDB with technical, aesthetic, and in-context branches; OmniVQA-MOS-20K, a human-labeled rating dataset; and OmniVQA-FG-Benchmark, a fine-grained understanding benchmark. The authors train a 7B 'Chatter' model for quality understanding and a 7B 'Rater' model for quality rating using a task-specific complementary training strategy. They report state-of-the-art results on Q-bench-video and on the new FG-Benchmark, and strong rating results on LSVQ, KoNViD-1k, YT-UGC, LIVE-VQC, and MOS-20K.

Significance. If the claims hold, the paper would make a useful contribution: it provides a large-scale, multi-branch VQA instruction dataset, a new fine-grained benchmark, and a complementary training recipe that shows clear gains over direct and mixed training. The rating results on external datasets (LSVQ, KoNViD-1k, YT-UGC, LIVE-VQC) are credible and provide independent support for the Rater. The data-scaling study in Fig. 7 is a valuable empirical finding. However, the central understanding claim is weakened by two load-bearing issues: the fine-grained benchmark's machine-annotated half is generated by the same synthetic-distortion pipeline as the training data, and the general-understanding comparison on Q-bench-video may not have been scored on a common filtered question set. The MIDB annotation chain also relies on a single in-house expert model with limited independent validation. These issues do not invalidate the rating contribution, but they require substantial revision before the state-of-the-art understanding claim can be accepted.

major comments (4)
  1. [Sec. 3.3 vs. Sec. 5] The machine-annotated half of the OmniVQA-FG-Benchmark is generated from the same candidate pool and the same synthetic spatial/temporal distortion scheme used to build the in-context training branch: source videos with objective quality above 70, a 1/4-frame rectangular region, 1-3 second durations, three intensity levels, and the same distortion-type/location set. Chatter achieves 69.87% versus VQA2-Assistant's 55.33% on machine items, but only 47.01% versus 46.50% on human-annotated items. Because the overall score (65.32% versus 54.12%) is dominated by the in-distribution machine portion, the claim of state-of-the-art fine-grained understanding is not established on independent data. Please report the human-only subset as the primary comparison, and either remove or clearly subordinate the machine-annotated part when drawing conclusions about generalization.
  2. [Sec. 7.3] The paper removes multi-video comparison questions from Q-bench-video because the training data do not include them, but it does not state whether every baseline was rescored on the identical filtered question subset. If the baseline numbers in Table 2 include multi-video questions, the comparison is not apples-to-apples. Please clarify the filtering procedure and, if necessary, rerun all models on the same filtered question set, reporting the number of questions removed.
  3. [Sec. 3.2 and Supp. Sec. E.2] The technical-branch annotations are produced by VQA2-Assistant (7B) - the authors' own prior model - followed by LLM summarization and sota-LMM voting, with human intervention only when a voting round receives a score of 0. No independent human agreement study is reported for the full 400K set, so the risk of systematic bias inherited from the expert/judge chain is not quantified. Because the quality-understanding state-of-the-art claim depends on the accuracy of these labels, please provide a human-annotation agreement analysis on a random sample of the technical branch, or an external validation showing that the machine annotations are not systematically biased relative to independent human perception.
  4. [Sec. 4] The MOS-20K collection protocol rejects any human score that deviates by two or more quality levels from an objective-quality reference derived from four objective VQA models. This makes the human ratings conditional on the objective estimators' ranking, so MOS-20K(test) is not an independent measure of rating performance and the MOS-20K column in Table 1 should be interpreted cautiously. The external rating datasets (LSVQ, KoNViD-1k, YT-UGC) provide independent support for the Rater, but the paper should nevertheless validate a subset of MOS-20K with unconstrained human ratings and report agreement between the filtered and unfiltered scores.
minor comments (6)
  1. [Sec. 5] The first sentence of Section 5 contains a typo: 'QmniVQA-FG-Benchmark' should be 'OmniVQA-FG-Benchmark'.
  2. [Table 7] The row labeled 'OmniVQA-MOS-400K' appears to be a naming error; the text consistently refers to 'OmniVQA-Chat-400K'. In addition, the row 'OmniVQA-Chat-20K' does not correspond to any dataset described in the paper and should be corrected or removed.
  3. [Abstract] The phrase 'the potential of scaling law remains unprecedented' is unclear; 'unprecedented' should likely be 'underexplored' or 'unverified'.
  4. [Sec. 3.2] 'Openai-o1' should be written consistently as 'OpenAI o1'.
  5. [Sec. 7.1] The statement 'we set almost unified system prompts' is vague; please specify which system-prompt components differ across models and evaluation settings.
  6. [Supp. Fig. 8] The subfigure captions in Fig. 8 appear misassigned: labels (d) and (e) are marked 'Technical Branch' but seem to describe the in-context branch, and subsequent labels are similarly offset. Please correct the caption-to-content mapping.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the machine-half overlap of the FG benchmark is an evaluation limitation, not a circular step.

full rationale

The paper's derivation chain is: construct a 400K MIDB via rejection sampling with VQA2-Assistant as expert plus human-in-the-loop voting and LLM summarization, construct OmniVQA-MOS-20K with human ratings, SFT the Chatter/Rater models, and evaluate on external rating datasets (LSVQ, KoNViD-1k, YT-UGC, LIVE-VQC), Q-bench-video, and OmniVQA-FG-Bench. None of the reported predictions is an algebraic transform of training labels or a fitted parameter renamed as a prediction; the external rating benchmarks are independent human-MOS tests, and the rating SOTA does not reduce to the training pipeline. Q-bench-video is from partially overlapping authors, and the paper filters out multi-video questions, which is a comparability caveat rather than a circular step. The machine-annotated half of OmniVQA-FG-Bench is generated from the same Sec. 3 candidate pool and distortion protocol as the in-context training branch, so the large machine-part margin (69.87 vs 55.33) is an in-distribution result and is weak as evidence of transfer; however, the benchmark answers are not used in training, and the paper separately reports the human-annotated half (47.01 vs 46.50), so this is an evaluation-design limitation, not an equation-level circularity. Self-citation to VQA2 is load-bearing as an annotation tool, but it is supplemented by human-in-the-loop voting and is not invoked as an external theorem; no uniqueness claim or ansatz is imported via self-citation. Overall, the central claims rest on independent rating benchmarks and on a transparently reported, if partially in-distribution, fine-grained evaluation.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a chain of machine-generated annotations and human-in-the-loop filters. The biggest burden comes from treating VQA2-Assistant and LMM judges as reliable annotators, and from using same-team benchmarks for understanding evaluation.

free parameters (5)
  • omega weights for quality-level scoring = [1, 0.75, 0.5, 0.25, 0]
    Hand-chosen weights in evaluation formula (Supp. A.2) that convert five quality-level logits into a predicted MOS; they affect all rating results.
  • N=5 expert samples per factor = 5
    Number of repeated questions asked of VQA2-Assistant in rejection sampling (Sec 3.2); hand-chosen.
  • 3-of-5 positive answer threshold = 3/5
    Threshold for an answer to be considered positive in LLM summary (Sec 3.2); hand-chosen.
  • objective quality threshold for branch selection = 70
    Threshold for selecting videos for in-context and aesthetic branches (Sec 3.3, 3.4); hand-chosen.
  • hidden-reference deviation threshold = 2 quality levels
    Any human rating differing by two or more levels from objective labels is rejected and rescored (Sec 4); hand-chosen.
assumptions (6)
  • domain assumption VQA2-Assistant's single-factor quality annotations are a reliable source for training supervision.
    Used as expert model in rejection sampling, Sec 3.2; if biased, the whole Chat-400K inherits bias.
  • domain assumption SOTA LMMs can accurately judge whether a quality-factor summary matches the video via voting.
    General LMMs act as judges in Sec 3.2; voting scores determine whether human experts must intervene.
  • domain assumption Averaged objective labels from four NR-VQA models provide a valid quality prior for stratification and for filtering human MOS.
    Used to build candidate pools (Sec 3.1) and as hidden reference in MOS-20K (Sec 4).
  • domain assumption Human scores deviating by two or more levels from the objective reference are annotation errors, not valid perceptions.
    Hidden-reference supervision, Sec 4; this shapes the ground-truth MOS distribution.
  • domain assumption Synthetic distortions (blur, exposure, noise, JPEG, stutter) approximate real distortions well enough to transfer to in-the-wild videos.
    In-context branch, Sec 3.3 and Supp. C.
  • domain assumption Q-bench-video is an independent external benchmark despite sharing authors and pipeline with the training data.
    Used as primary understanding benchmark in Sec 7.3; same research group authored Q-bench-video [57].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling-up Perceptual Video Quality Assessment." pith.science (2026). https://pith.science/paper/QF7XK2PE

@misc{pith2026250522543,
  author       = {Pith},
  title        = {Pith review of: Scaling-up Perceptual Video Quality Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QF7XK2PE}},
  note         = {Machine review of arXiv:2505.22543}
}
read the original abstract

The data scaling law has been shown to significantly enhance the performance of large multi-modal models (LMMs) across various downstream tasks. However, in the domain of perceptual video quality assessment (VQA), the potential of scaling law remains unprecedented due to the scarcity of labeled resources and the insufficient scale of datasets. To address this, we propose \textbf{OmniVQA}, an efficient framework designed to efficiently build high-quality, human-in-the-loop VQA multi-modal instruction databases (MIDBs). We then scale up to create \textbf{OmniVQA-Chat-400K}, the largest MIDB in the VQA field concurrently. Our focus is on the technical and aesthetic quality dimensions, with abundant in-context instruction data to provide fine-grained VQA knowledge. Additionally, we have built the \textbf{OmniVQA-MOS-20K} dataset to enhance the model's quantitative quality rating capabilities. We then introduce a \textbf{complementary} training strategy that effectively leverages the knowledge from datasets for quality understanding and quality rating tasks. Furthermore, we propose the \textbf{OmniVQA-FG (fine-grain)-Benchmark} to evaluate the fine-grained performance of the models. Our results demonstrate that our models achieve state-of-the-art performance in both quality understanding and rating tasks.

Figures

Figures reproduced from arXiv: 2505.22543 by the authors.

Figure 1
Figure 1. OmniVQA overview. On the left side, it summarizes the existing MIDB construction paradigms in the visual quality assessment field. The main drawbacks of these methods lie in their excessive reliance on manual annotations or existing datasets, as well as the lack of comprehensive and diverse annotation dimensions. To address this, we have developed the OmniVQA datasets and models. On the right side, it shows examples… view at source ↗
Figure 2
Figure 2. Data construction pipeline of OmniVQA-Chat-400K. nical [17, 46–48, 52, 53] or aesthetic quality [16, 59] of im￾ages/videos while lacking comprehensive annotations inte￾grating multiple quality dimensions and factors. Moreover, the majority of these MIDBs [16, 17, 46–48, 52, 53, 59] emphasize an overall description or understanding of im￾age/video quality but lack spatiotemporal fine-grained an￾notation and evaluatio… view at source ↗
Figure 3
Figure 3. Statistical information of OmniVQA-Chat-400K and OmniVQA-MOS-20K . with reasoning LLMs, and a voting mechanism using gen￾eral LMMs is used to decide the next process. In this case, the general LMMs act as a judger rather than an annotator. We argue that substituting the challenging direct annotation task with the relatively simpler voting-based decision for general LMMs offers a more practical distillation approach.… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Statistical information of the OmniVQA-FG-Bench. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Illustration of OmniVQA models and the complemen [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The performance map of scaling up the training dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Additional Statistical Information [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Spatial-distorted videos examples [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Annotation prompts for technical branch (1). [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Annotation prompts for technical branch (2). [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Annotation prompts for technical branch (3). [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Annotation prompts for in-context branch [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Annotation prompts for aesthetic branch. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: Technical branch annotation example(1). (Note that the “summary and refine” process is done after the human-in-the-loop [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Technical-branch annotation example(2). (Note that the “summary and refine” process is done after the human-in-the-loop [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Technical branch annotation example(3) [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Aesthetic branch annotation examples [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: In-context branch annotation examples. Since frame freezing effects cannot be demonstrated through screenshots, we provide [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]
Figure 20
Figure 20. Figure 20: Subjective Quality Rating Experiment Interface. This interface is designed for the subjective quality rating experiment. The annotation workflow consists of 3 explicit phases. Firstly, the annotator inputs their assigned group number into the designated textbox (top-c…
Figure 21
Figure 21. Figure 21: Human-in-the-loop Selection Experiment Interface.This interface facilitates a human-in-the-loop approach to video quality assessment. The annotation workflow is composed of three phases. Firstly, the annotator enters their assigned group number into the first designat…
Figure 22
Figure 22. Figure 22: This is a human annotation interface with the following structure: The first row displays [Question], where the annotator fills in [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: Examples of questions and correct answers of the OmniVQA-FG-Benchmark [PITH_FULL_IMAGE:figures/full_fig_p031_23.png]
Figure 24
Figure 24. Figure 24: Examples of questions and correct answers of the OmniVQA-FG-Benchmark [PITH_FULL_IMAGE:figures/full_fig_p032_24.png]
Figure 25
Figure 25. Figure 25: This is a self-recorded video excerpt from the LBVD dataset. The video exhibits severe overexposure in the upper right corner [PITH_FULL_IMAGE:figures/full_fig_p033_25.png]
Figure 26
Figure 26. Figure 26: This is a smartphone-recorded video. For most of the playback duration, the camera remains stable; however, at the beginning [PITH_FULL_IMAGE:figures/full_fig_p034_26.png]
Figure 27
Figure 27. Figure 27: This video was recorded using a professional camera with background defocus processing to enhance its aesthetic appeal. [PITH_FULL_IMAGE:figures/full_fig_p035_27.png]
Figure 28
Figure 28. Figure 28: This video is manually recorded using a smartphone, with varying focal points over time. At the beginning of the video, the [PITH_FULL_IMAGE:figures/full_fig_p036_28.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 63 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 7

    arXiv 2025

  3. [3]

    Feature-based predic- tion of streaming video qoe: Distortions, stalling and mem- ory.SPIC, 68:218–228, 2018

    Christos G Bampis and Alan C Bovik. Feature-based predic- tion of streaming video qoe: Distortions, stalling and mem- ory.SPIC, 68:218–228, 2018. 2

    work page 2018

  4. [4]

    Towards perceptually optimized adaptive video streaming-a realistic quality of experience database.IEEE TIP, 30:5182–5197,

    Christos G Bampis, Zhi Li, Ioannis Katsavounidis, Te-Yuan Huang, Chaitanya Ekanadham, and Alan C Bovik. Towards perceptually optimized adaptive video streaming-a realistic quality of experience database.IEEE TIP, 30:5182–5197,

    work page

  5. [5]

    Q-ground: Image quality grounding with large multi-modality models

    Chaofeng Chen, Sensen Yang, Haoning Wu, Liang Liao, Zicheng Zhang, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Q-ground: Image quality grounding with large multi-modality models. InACM MM, pages 486–495, 2024. 2

    work page 2024

  6. [6]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. InCVPR, pages 13320–13331,

  7. [7]

    Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test- time scaling.arXiv preprint arXiv:2412.05271, 2024. 7

    arXiv 2024

  8. [8]

    Grounding-iqa: Multimodal lan- guage grounding model for image quality assessment.arXiv preprint arXiv:2411.17237, 2024

    Zheng Chen, Xun Zhang, Wenbo Li, Renjing Pei, Feng- long Song, Xiongkuo Min, Xiaohong Liu, Xin Yuan, Yong Guo, and Yulun Zhang. Grounding-iqa: Multimodal lan- guage grounding model for image quality assessment.arXiv preprint arXiv:2411.17237, 2024. 2

    arXiv 2024

Show all 79 references
  1. [9]

    A quality-of-experience index for streaming video.IEEE JSTSP, 11(1):154–166, 2016

    Zhengfang Duanmu, Kai Zeng, Kede Ma, Abdul Rehman, and Zhou Wang. A quality-of-experience index for streaming video.IEEE JSTSP, 11(1):154–166, 2016. 2

    2016

  2. [10]

    A quality-of-experience database for adaptive video streaming

    Zhengfang Duanmu, Abdul Rehman, and Zhou Wang. A quality-of-experience database for adaptive video streaming. IEEE TBC, 64(2):474–487, 2018. 2

    2018

  3. [11]

    A bayesian quality-of-experience model for adaptive streaming videos

    Zhengfang Duanmu, Wentao Liu, Diqi Chen, Zhuoran Li, Zhou Wang, Yizhou Wang, and Wen Gao. A bayesian quality-of-experience model for adaptive streaming videos. ACM TOMM, 18(3s):1–24, 2023. 2

    2023

  4. [12]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6202–6211, 2019. 6

    2019

  5. [13]

    Lmm-vqa: Advancing video quality assessment with large multimodal models.arXiv preprint arXiv:2408.14008,

    Qihang Ge, Wei Sun, Yu Zhang, Yunhao Li, Zhongpeng Ji, Fengyu Sun, Shangling Jui, Xiongkuo Min, and Guangtao Zhai. Lmm-vqa: Advancing video quality assessment with large multimodal models.arXiv preprint arXiv:2408.14008,

    arXiv

  6. [14]

    In- capture mobile video distortions: A study of subjective be- havior and objective algorithms.IEEE TCSVT, 28(9):2061– 2077, 2017

    Deepti Ghadiyaram, Janice Pan, Alan C Bovik, Anush Kr- ishna Moorthy, Prasanjit Panda, and Kai-Chieh Yang. In- capture mobile video distortions: A study of subjective be- havior and objective algorithms.IEEE TCSVT, 28(9):2061– 2077, 2017. 2

    2017

  7. [15]

    The konstanz natural video database (konvid-1k)

    Vlad Hosu, Franz Hahn, Mohsen Jenadeleh, Hanhe Lin, Hui Men, Tam´as Szir ´anyi, Shujun Li, and Dietmar Saupe. The konstanz natural video database (konvid-1k). InQoMEX, pages 1–6. IEEE, 2017. 7

    2017

  8. [16]

    Aesexpert: Towards multi-modality foun- dation model for image aesthetics perception

    Yipo Huang, Xiangfei Sheng, Zhichao Yang, Quan Yuan, Zhichao Duan, Pengfei Chen, Leida Li, Weisi Lin, and Guangming Shi. Aesexpert: Towards multi-modality foun- dation model for image aesthetics perception. InACM MM, pages 5911–5920, 2024. 2, 3, 5

    2024

  9. [17]

    Vqa2: Visual question answering for video quality assessment.arXiv preprint arXiv:2411.03795,

    Ziheng Jia, Zicheng Zhang, Jiaying Qian, Haoning Wu, Wei Sun, Chunyi Li, Xiaohong Liu, Weisi Lin, Guangtao Zhai, and Xiongkuo Min. Vqa2: Visual question answering for video quality assessment.arXiv preprint arXiv:2411.03795,

    arXiv

  10. [18]

    Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large lan- guage models.arXiv preprint arXiv:2402.10038, 2024

    Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ra- machandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large lan- guage models.arXiv preprint arXiv:2402.10038, 2024. 3

    2024 arXiv

  11. [19]

    Two-level approach for no-reference con- sumer video quality assessment.IEEE TIP, 28(12):5923– 5938, 2019

    Jari Korhonen. Two-level approach for no-reference con- sumer video quality assessment.IEEE TIP, 28(12):5923– 5938, 2019. 2

    2019

  12. [20]

    Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception.IEEE TCSVT, 32(9):5944–5958, 2022

    Bowen Li, Weixia Zhang, Meng Tian, Guangtao Zhai, and Xianpei Wang. Blindly assess quality of in-the-wild videos via quality-aware pre-training and motion perception.IEEE TCSVT, 32(9):5944–5958, 2022. 2, 7

    2022

  13. [21]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 7

    2024 arXiv

  14. [22]

    Quality as- sessment of in-the-wild videos

    Dingquan Li, Tingting Jiang, and Ming Jiang. Quality as- sessment of in-the-wild videos. InACM MM, pages 2351– 2359, 2019. 2

    2019

  15. [23]

    A comparative study of state-of-the-art video encoders on 4k videos.Image Anal- ysis and Recognition

    Z Li, Z Duanmu, W Liu, and Z Wang. A comparative study of state-of-the-art video encoders on 4k videos.Image Anal- ysis and Recognition. LNCS, 11662. 2

  16. [24]

    Statistical rejec- tion sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mo- hammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejec- tion sampling improves preference optimization. InICLR. 3

  17. [25]

    End- to-end blind quality assessment of compressed videos using deep neural networks

    Wentao Liu, Zhengfang Duanmu, and Zhou Wang. End- to-end blind quality assessment of compressed videos using deep neural networks. InACM MM, pages 546–554, 2018. 2

    2018

  18. [26]

    Perceptual video quality assessment: A sur- vey.SCIS, 67(11):211301, 2024

    Xiongkuo Min, Huiyu Duan, Wei Sun, Yucheng Zhu, and Guangtao Zhai. Perceptual video quality assessment: A sur- vey.SCIS, 67(11):211301, 2024. 2

    2024

  19. [27]

    No-reference image quality assessment in the spa- tial domain.IEEE TIP, 21(12):4695–4708, 2012

    Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment in the spa- tial domain.IEEE TIP, 21(12):4695–4708, 2012. 2

    2012

  20. [28]

    completely blind

    Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Mak- ing a “completely blind” image quality analyzer.IEEE SPL, 20(3):209–212, 2012. 2

    2012

  21. [29]

    Cvd2014—a database for evaluating no-reference video quality assess- ment algorithms.IEEE TIP, 25(7):3073–3086, 2016

    Mikko Nuutinen, Toni Virtanen, Mikko Vaahteranoksa, Tero Vuori, Pirkko Oittinen, and Jukka H ¨akkinen. Cvd2014—a database for evaluating no-reference video quality assess- ment algorithms.IEEE TIP, 25(7):3073–3086, 2016. 2

    2016

  22. [30]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 2

    2018

  23. [31]

    Large-scale study of perceptual video quality.IEEE TIP, 28(2):612–627, 2018

    Zeina Sinno and Alan Conrad Bovik. Large-scale study of perceptual video quality.IEEE TIP, 28(2):612–627, 2018. 2, 7

    2018

  24. [32]

    A deep learning based no-reference quality assessment model for ugc videos

    Wei Sun, Xiongkuo Min, Wei Lu, and Guangtao Zhai. A deep learning based no-reference quality assessment model for ugc videos. InACM MM, pages 856–865, 2022. 2, 7

    2022

  25. [33]

    Analysis of video quality datasets via design of minimalistic video quality models.IEEE TPAMI,

    Wei Sun, Wen Wen, Xiongkuo Min, Long Lan, Guangtao Zhai, and Kede Ma. Analysis of video quality datasets via design of minimalistic video quality models.IEEE TPAMI,

  26. [34]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024. 2, 7

    2024 arXiv

  27. [35]

    Ugc-vqa: Benchmarking blind video quality assessment for user generated content.IEEE TIP, 30:4449–4464, 2021

    Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, and Alan C Bovik. Ugc-vqa: Benchmarking blind video quality assessment for user generated content.IEEE TIP, 30:4449–4464, 2021. 2

    2021

  28. [36]

    Rapique: Rapid and accurate video quality prediction of user generated content

    Zhengzhong Tu, Xiangxu Yu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, and Alan C Bovik. Rapique: Rapid and accurate video quality prediction of user generated content. IEEE OJSP, 2:425–440, 2021. 2

    2021

  29. [37]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024. 7

    2024 arXiv

  30. [38]

    Youtube ugc dataset for video compression research

    Yilin Wang, Sasi Inguva, and Balu Adsumilli. Youtube ugc dataset for video compression research. InIEEE MMSP, pages 1–5. IEEE, 2019. 2, 7

    2019

  31. [39]

    Rich features for perceptual quality assessment of ugc videos

    Yilin Wang, Junjie Ke, Hossein Talebi, Joong Gon Yim, Neil Birkbeck, Balu Adsumilli, Peyman Milanfar, and Feng Yang. Rich features for perceptual quality assessment of ugc videos. InCVPR, pages 13435–13444, 2021. 2

    2021

  32. [40]

    Llm- generated natural language meets scaling laws: New ex- plorations and data augmentation methods.arXiv preprint arXiv:2407.00322, 2024

    Zhenhua Wang, Guang Xu, and Ming Ren. Llm- generated natural language meets scaling laws: New ex- plorations and data augmentation methods.arXiv preprint arXiv:2407.00322, 2024. 1

    2024 arXiv

  33. [41]

    Modular blind video quality assess- ment

    Wen Wen, Mu Li, Yabin Zhang, Yiting Liao, Junlin Li, Li Zhang, and Kede Ma. Modular blind video quality assess- ment. InCVPR, pages 2763–2772, 2024. 2, 7

    2024

  34. [42]

    Neigh- bourhood representative sampling for efficient end-to-end video quality assessment.IEEE TPAMI, 45(12):15185– 15202, 2023

    Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, Jinwei Gu, and Weisi Lin. Neigh- bourhood representative sampling for efficient end-to-end video quality assessment.IEEE TPAMI, 45(12):15185– 15202, 2023. 3, 7

    2023

  35. [43]

    Discovqa: Temporal distortion-content transformers for video quality assessment

    Haoning Wu, Chaofeng Chen, Liang Liao, Jingwen Hou, Wenxiu Sun, Qiong Yan, and Weisi Lin. Discovqa: Temporal distortion-content transformers for video quality assessment. IEEE TCSVT, 33(9):4840–4854, 2023

    2023

  36. [44]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives

    Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jing- wen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. InICCV, pages 20144–20154, 2023. 3, 7

    2023

  37. [45]

    Towards explainable in-the-wild video quality assess- ment: A database and a language-prompted approach

    Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jing- wen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Towards explainable in-the-wild video quality assess- ment: A database and a language-prompted approach. In ACM MM, pages 1045–1054, 2023. 2

    2023

  38. [46]

    Q-instruct: Improving low-level visual abilities for multi-modality foundation models

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al. Q-instruct: Improving low-level visual abilities for multi-modality foundation models. In CVPR, pages 25490–25500, 2024. 2, 3

    2024

  39. [47]

    Q-align: teaching lmms for visual scoring via discrete text-defined levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: teaching lmms for visual scoring via discrete text-defined levels. InICML, pages 54015–54029, 2024. 2, 3, 7

    2024

  40. [48]

    Towards open-ended vi- sual quality comparison

    Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, et al. Towards open-ended vi- sual quality comparison. InECCV, pages 360–377. Springer,

  41. [49]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 6

    2024 arXiv

  42. [50]

    mplug- owl3: Towards long image-sequence understanding in multi- modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug- owl3: Towards long image-sequence understanding in multi- modal large language models. InICLR. 7

  43. [51]

    Patch-vq:’patching up’the video quality problem

    Zhenqiang Ying, Maniratnam Mandal, Deepti Ghadiyaram, and Alan Bovik. Patch-vq:’patching up’the video quality problem. InCVPR, pages 14019–14029, 2021. 2, 5, 7

    2021

  44. [52]

    Descriptive image quality assessment in the wild.arXiv preprint arXiv:2405.18842,

    Zhiyuan You, Jinjin Gu, Zheyuan Li, Xin Cai, Kaiwen Zhu, Chao Dong, and Tianfan Xue. Descriptive image quality assessment in the wild.arXiv preprint arXiv:2405.18842,

  45. [53]

    Depicting beyond scores: Advanc- ing image quality assessment through multi-modal language models

    Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, and Chao Dong. Depicting beyond scores: Advanc- ing image quality assessment through multi-modal language models. InECCV, pages 259–276. Springer, 2024. 2, 3

    2024

  46. [54]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV, pages 11975–11986, 2023. 6

    2023

  47. [55]

    When scaling meets llm finetuning: The effect of data, model and finetuning method

    Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets llm finetuning: The effect of data, model and finetuning method. InICLR, 2024. 1

    2024

  48. [56]

    Video instruction tuning with synthetic data.CoRR, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.CoRR, 2024. 1

    2024

  49. [57]

    Q-bench-video: Benchmarking the video quality understanding of lmms.CVPR, 2025

    Zicheng Zhang, Ziheng Jia, Haoning Wu, Chunyi Li, Zijian Chen, Yingjie Zhou, Wei Sun, Xiaohong Liu, Xiongkuo Min, Weisi Lin, et al. Q-bench-video: Benchmarking the video quality understanding of lmms.CVPR, 2025. 2

    2025

  50. [58]

    Lima: Less is more for alignment.NIPS, 36:55006– 55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.NIPS, 36:55006– 55021, 2023. 1

    2023

  51. [59]

    Uniaa: A unified multi-modal image aesthetic assessment baseline and benchmark.arXiv preprint arXiv:2404.09619,

    Zhaokun Zhou, Qiulin Wang, Bin Lin, Yiwei Su, Rui Chen, Xin Tao, Amin Zheng, Li Yuan, Pengfei Wan, and Di Zhang. Uniaa: A unified multi-modal image aesthetic assessment baseline and benchmark.arXiv preprint arXiv:2404.09619,

    arXiv

  52. [61]

    For instance, if the bounding box contains a part of a human body while the surrounding area shows a background wall, this would be valuable for annotation

    The content inside the bounding box should exhibit a significant semantic difference from the surrounding area outside the bounding box. For instance, if the bounding box contains a part of a human body while the surrounding area shows a background wall, this would be valuable...

  53. [62]

    Additionally, the object should occupy a relatively large proportion of the bounding box

    The content inside the bounding box should ideally contain a complete, describable object, such as a human face, the body of a person, or the right half of a face. Additionally, the object should occupy a relatively large proportion of the bounding box

  54. [63]

    the left half of the face

    The content inside the bounding box should maintain temporal consistency, meaning there should be no significant semantic changes or scene transitions within the sequence of frames containing the bounding box. If all three criteria are met, the region is considered annotatable...

  55. [64]

    Ask and answer the specific start and end seconds or the duration of the spatiotemporal local distortion in the video

  56. [65]

    If only a general description of the location is provided (such as center, top left, bottom right, etc.), answer with a location description

    Ask and answer the location of the spatiotemporal local distortion within the video frame. If only a general description of the location is provided (such as center, top left, bottom right, etc.), answer with a location description. If the description contains specific semanti...

  57. [66]

    The answer should only use the information in the descriptions without adding any additional message

    Ask what specific distortion occurred at a particular location and for a specific object (if there is semantic information provided), and inquire about the severity of the distortion. The answer should only use the information in the descriptions without adding any additional ...

  58. [69]

    Question Answer

    Open-ended question: Format: "Question Answer". Please ensure that the four different question-answer pairs have diverse question formats and you don't have to set the three types of questions in order, you can shuffle them in any sequences. Final output should be in the follo...

  59. [70]

    Please aim to be concise, yet accurate and professional in your description

    First, summarize the overall aesthetic style of the video based on the keyframe sampling sequence. Please aim to be concise, yet accurate and professional in your description. (SPATIAL-TEMPORAL ANALYSIS)

  60. [71]

    Spatial: Provide a brief analysis of the compositional beauty and rationality, clarity of the main subject, and the comfort and accuracy of the main content

    In the second step, analyze the aesthetic effects of the video from the following two perspectives: a. Spatial: Provide a brief analysis of the compositional beauty and rationality, clarity of the main subject, and the comfort and accuracy of the main content. Please analyze t...

  61. [72]

    "" Step 2: Instruction Generation System Prompt:

    Finally, provide a precise one-sentence description of the human emotional feedback the video evokes (i.e., what kind of feeling it gives to human) (Please note that the emotional response evoked by the video should not only include the emotional feedback arising from its cont...

  62. [73]

    Inquire and answer the overall style of this video

  63. [74]

    Inquire and answer the emotional feedback this video evokes

  64. [75]

    The question formats should be as follows:

    Inquire and answer a specific aesthetic characteristic regarding the spatial or temporal aspect of this video. The question formats should be as follows:

  65. [76]

    Question + A. Option one, B. Option two Answer: correct choice + answer

    Yes/No question (with two options, single answer): Format: "Question + A. Option one, B. Option two Answer: correct choice + answer"

  66. [77]

    Question + A. Option one, B. Option two, C. Option three, D. Option four Answer: correct choice + answer

    Multiple choice (four options, single answer): Format: "Question + A. Option one, B. Option two, C. Option three, D. Option four Answer: correct choice + answer"

  67. [78]

    Question Answer

    Open-ended question: Format: "Question Answer". Please ensure that the four different question-answer pairs have diverse question formats and you don't have to set the three types of questions in order, you can shuffle them in any sequences. Final output should be in the follo...

  68. [79]

    , "prefix

    Aesthetic Effects: a. **Spatial:** The composition is straightforward, with two subjects seated in profile, creating a balanced, symmetrical feel. The subjects are the clear focus, with minimal distracting background elements. The framing is comfortable, avoiding extreme close...

  69. [80]

    **Spatial:** The composition is straightforward, offering a clear, wide-angle perspective of the fields

    **Aesthetic Effects Analysis:** a. **Spatial:** The composition is straightforward, offering a clear, wide-angle perspective of the fields. The main subject, the patterned agricultural land, is easily discernible. The framing offers a comfortable view, accurately depicting the...

  70. [81]

    , "prefix

    **Emotional Feedback and Overall Aesthetic Effect:** The video evokes a feeling of tranquility and spaciousness. The overall aesthetic effect is fair.", "prefix": "Now you will receive one video. This video is 6 seconds long, and you will see a sequence of images generated by ...

  71. [2024]

    In addition, you will receive a motion feature sequence that corresponds to the number of frames in the video, num of frames

    2, 3 Scaling-up Perceptual Video Quality Assessment Supplementary Material A. Experiments Supplementary Materials A.1. System prompts for training and evaluation In all training process and task evaluations, we set uni- fied system prompts (prefix) for all LMM models, which is...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.