REVIEW 2 major objections 5 minor 40 references
A 847-clip near-raw talking-head webcam dataset shows that content type and background processing change codec efficiency, with H.266 saving up to 71.3% VMAF BD-rate versus H.264.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-13 20:12 UTC pith:VWWOSFC3
load-bearing objection Solid, carefully documented near-raw talking-head corpus that is 5× larger than the prior webcam set and actually useful for codec and SR work once released; residual camera MJPEG is disclosed, not hidden. the 2 major comments →
A Camera-Native Talking-Head Video Dataset for Various Computer Vision Tasks
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
A camera-native, losslessly stored talking-head corpus of 847 clips, together with a stratified 120-clip benchmark that includes original, blurred-background, and replaced-background versions, is large and clean enough to expose statistically significant encoder-by-dataset and encoder-by-content-condition interactions; modern codecs therefore yield large VMAF BD-rate savings (up to −71.3% for H.266 versus H.264) whose magnitude depends on both the content type and the background processing applied.
What carries the argument
Near-raw capture pipeline: each webcam is opened at its highest supported resolution and preferred pixel format (YUYV/NV12 when available, MJPEG otherwise), frames are stored with the mathematically lossless FFV1 codec, and a stratified 120-clip subset is drawn across MOS, spatial-temporal complexity, and three content conditions (original / background blur / background replacement).
Load-bearing premise
The residual processing that happens inside consumer webcam firmware—demosaicing, white-balance, auto-exposure, and the fact that three-quarters of the clips already arrive MJPEG-compressed from the camera—does not itself introduce artifacts that would undermine the claim of a clean, camera-native reference.
What would settle it
Re-encode the same 120-clip benchmark with an identical codec suite after deliberately adding a second controlled lossy stage that mimics typical capture-software compression; if the reported encoder-by-dataset and encoder-by-content interactions and the −71.3% H.266 savings disappear or reverse, the near-raw claim fails.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a near-raw talking-head webcam dataset of 847 clips (15 s each, ~212 min total) from 805 participants and 446 unique consumer cameras, stored with FFV1 lossless encoding so that the camera-native signal (24.4 % uncompressed YUYV/NV12, 75.6 % camera MJPEG) is preserved without additional lossy capture-pipeline compression. Each clip carries an ACR MOS (ITU-T P.910) and ten multi-label perceptual quality tokens that jointly explain 64.4 % of MOS variance; a stratified 120-clip benchmarking subset is released in three content conditions (original, background blur, background replacement). Codec experiments on four datasets and four encoders (H.264, H.265, H.266, AV1) report VMAF BD-rate savings up to −71.3 % for H.266 relative to H.264, together with significant encoder×dataset (η_p² = .112) and encoder×content-condition (η_p² = .149) interactions; a preliminary super-resolution evaluation is also sketched. The central claim is that the corpus and stratified benchmark constitute a useful public resource for compression, SR, quality assessment and enhancement research in real-time communication.
Significance. If the data are released as described, the work supplies a domain-specific, higher-fidelity talking-head resource that is five times larger than the previous largest public webcam set (VCD, 160 clips) and avoids the double-compression artifacts that have limited earlier corpora. The multi-phase MOS/token annotation with cross-study reliability (Pearson r ≥ 0.859), the transparent stratification procedure, and the reporting of BD-rates with confidence intervals and partial-eta-squared effect sizes are concrete strengths that make the resource immediately usable for codec and enhancement benchmarking. The encoder×dataset and encoder×content-condition interactions are of practical interest for RTC codec design. The residual camera-firmware processing (including the 75.6 % MJPEG fraction) is disclosed rather than hidden, so the comparative claim relative to prior lossy webcam datasets remains well-supported.
major comments (2)
- The abstract and introduction advertise a preliminary super-resolution evaluation with four SR models that “confirms that the dataset significantly affects absolute performance while preserving model rankings.” The body of the manuscript (as provided) contains no corresponding section, table, model list, or quantitative results. Either the SR experiment must be fully reported (models, downsampling protocol, metrics, statistical tests) or the claim must be removed from the abstract and contribution list; otherwise the paper over-promises relative to what is shown.
- Appendix C / Table VI is left as a skeleton of placeholders (“TH ���������������������������”). The per-group mean BD-rates and 95 % confidence intervals that are invoked to support the encoder×content-condition interaction (η_p² = .149) are therefore not inspectable. The numerical values (or a complete table) must be restored so that the interaction claim can be verified.
minor comments (5)
- Title inconsistency: the arXiv title uses “Camera-Native” while the manuscript header uses “Near-Raw.” Align terminology throughout.
- Table I percentages are stated to be independently rounded; a short note that they may not sum to 100 % is already present, but the same disclaimer should appear under any subsequent percentage tables.
- Figure 5 rate–distortion curves are referenced but the caption and axis labels in the source are incomplete; ensure final figures include bpp units and codec legends that match the text.
- The free parameters of the stratification (MOS bin edges, SI×TI medians, 25/50/25 target, token-coverage threshold ≥2) are reasonable but should be listed once in a single “reproducibility” paragraph so that future users can regenerate the 120-clip subset.
- A few typographic issues remain (e.g., “A V1” spacing, “�” characters in correlation symbols). A final proof-reading pass is needed.
Circularity Check
No circularity: empirical dataset paper with measured BD-rates and MOS annotations; self-citations to prior VCD are comparative only.
full rationale
This is a dataset-and-benchmark paper, not a first-principles derivation. The load-bearing claims are (i) collection of 847 near-raw talking-head clips stored with FFV1, (ii) ACR MOS plus ten quality tokens with reported reliability (Pearson r ≥ 0.859) and R² = 0.644 of MOS, (iii) a stratified 120-clip subset, and (iv) measured codec RD curves yielding VMAF BD-rate savings and ANOVA interactions (η_p² = .112, .149). None of these quantities is defined in terms of the others, fitted then re-presented as a prediction, or forced by a uniqueness theorem. Self-citations to the authors’ earlier VCD dataset [2] appear only as a scale/fidelity baseline (160 lossy clips vs. 847 near-raw) and as reference material in the token annotation study; they do not supply premises that make the new BD-rate or interaction results true by construction. External metrics (VMAF, PSNR, Bjøntegaard BD-rate, ITU-T P.910) and standard encoders (H.264/5/6, AV1) are used without smuggled ansätze. The residual camera-firmware/MJPEG caveat is a disclosed fidelity limitation, not a circular reduction. Score 0 is therefore appropriate.
Axiom & Free-Parameter Ledger
free parameters (4)
- MOS bin boundaries
- SI×TI quadrant split
- token selection threshold
- target MOS distribution 25/50/25
axioms (4)
- domain assumption ITU-T Rec. P.910 Absolute Category Rating yields a valid Mean Opinion Score for talking-head video
- domain assumption Bjøntegaard BD-rate computed on VMAF/PSNR is a valid measure of codec efficiency
- standard math FFV1 Level 3 is bit-exact lossless
- ad hoc to paper Camera firmware demosaicing/white-balance/gamma are acceptable residual processing for a ‘near-raw’ reference
invented entities (1)
-
ten perceptual quality tokens taxonomy
no independent evidence
read the original abstract
Talking-head videos constitute a predominant content type in real-time communication, yet publicly available datasets for video processing research in this domain remain scarce and limited in signal fidelity. In this paper, we open-source a camera-native dataset of 847 talking-head recordings (approximately 212 minutes), each 15s in duration, captured from 805 participants using 446 unique consumer webcam devices in their natural environments. All recordings are stored using the FFV1 lossless codec, preserving the camera-native signal -- uncompressed (24.4%) or MJPEG-encoded (75.6%) -- without additional lossy processing. Each recording is annotated with a Mean Opinion Score (MOS) and ten perceptual quality tokens that jointly explain 64.4% of the MOS variance. From this corpus, we curate a stratified benchmarking subset of 120 clips in three content conditions: original, background blur, and background replacement. Codec efficiency evaluation across four datasets and four codecs, namely H.264, H.265, H.266, and AV1, yields VMAF BD-rate savings up to $-71.3\%$ (H.266) relative to H.264, with significant encoder$\times$dataset ($\eta_p^2 = .112$) and encoder$\times$content condition ($\eta_p^2 = .149$) interactions, demonstrating that both content type and background processing affect compression efficiency. A preliminary super-resolution evaluation with four SR models confirms that the dataset significantly affects absolute performance while preserving model rankings, demonstrating applicability beyond codec benchmarking. The dataset offers 5$\times$ the scale of the largest prior talking-head webcam dataset (847 vs. 160 clips) with lossless signal fidelity, establishing a resource for benchmarking video compression, super-resolution, quality assessment, and enhancement models in real-time communication.
Reference graph
Works this paper leans on
-
[1]
near-raw,
serve as widely adopted baselines. Real-world SR datasets address the realism gap: RealVSR [19] captures paired se- quences with two cameras at different focal lengths, and VideoLQ [20] provides a benchmark for blind real-world video SR. However, these datasets focus on general scenes and do not capture the domain-specific characteristics of webcam talkin...
-
[2]
A large-scale, near-raw talking-head webcam video dataset comprising 847 losslessly encoded recordings from 805 participants across 446 unique camera configu- rations, with four recording scenarios covering common video conferencing behaviors
-
[3]
A multi-dimensional quality annotation scheme com- bining ACR-based MOS with ten perceptual quality tokens, validated through cross-study reliability analysis (Pearson�≥0�859between independent annotation studies)
-
[4]
A stratified benchmarking subset of 120 clips in three groups (TH, TH-BB, TH-BR), balanced across quality levels, spatial–temporal complexity, and distortion types
-
[5]
near-raw
An evaluation of the dataset’s utility for codec com- pression efficiency analysis across H.264 (A VC) [24], H.265 (HEVC) [25], H.266 (VVC) [26], and A V1 [27] (see Section III). II. DATASET This section describes the data collection methodology, the composition of the published dataset, the quality annotation process, and the construction of the benchmar...
1920
-
[6]
Stratification strategy.:The subset selection employs a stratified sampling algorithm to ensure that each group covers the full range of quality levels, spatial–temporal complexity, and distortion types. Clips are assigned to one of 12 strata formed by the Cartesian product of three MOS bins (Low: [1�0�2�8), Medium:[2�8�4�0), High:[4�0�5�0]) and four SI×T...
2022
-
[7]
VCD: A Video Conferencing Dataset for Video Compression,
B. Naderi, R. Cutler, N. S. Khongbantabam, Y . Hosseinkashi, H. Turbell, A. Sadovnikov, and Q. Zou, “VCD: A Video Conferencing Dataset for Video Compression,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 3970–3974
2024
-
[8]
VFHQ: A high-quality dataset and benchmark for video face super-resolution,
L. Xie, X. Wang, H. Zhang, C. Dong, and Y . Shan, “VFHQ: A high-quality dataset and benchmark for video face super-resolution,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022, pp. 657–666
2022
-
[9]
Common test conditions and software reference configura- tions,
F. Bossen, “Common test conditions and software reference configura- tions,”JCTVC-L1100, vol. 12, no. 7, 2013
2013
-
[10]
JVET common test conditions and software reference configurations for SDR video,
F. Bossen, J. Boyce, X. Li, V . Seregin, and K. Suhring, “JVET common test conditions and software reference configurations for SDR video,” Joint Video Experts Team (JVET) of ITU-T SG�, vol. 16, pp. 19–27, 2019
2019
-
[11]
UVG dataset: 50/120fps 4K sequences for video codec analysis and development,
A. Mercat, M. Viitanen, and J. Vanne, “UVG dataset: 50/120fps 4K sequences for video codec analysis and development,” in Proceedings of the 11th ACM Multimedia Systems Conference. Istanbul Turkey: ACM, May 2020, pp. 297–302. [Online]. Available: https://dl.acm.org/doi/10.1145/3339825.3394937
-
[12]
MCL-JCV: A JND- based H.264/A VC video quality assessment dataset,
H. Wang, W. Gan, S. Hu, J. Y . Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “MCL-JCV: A JND- based H.264/A VC video quality assessment dataset,” in2016 IEEE International Conference on Image Processing (ICIP), Sep. 2016, pp. 1509–1513, iSSN: 2381-8549
2016
-
[13]
V oxCeleb: a large-scale speaker identification dataset,
A. Nagrani, J. S. Chung, and A. Zisserman, “V oxCeleb: a large-scale speaker identification dataset,”arXiv:1706.08612 [cs], Jun. 2017, arXiv: 1706.08612. [Online]. Available: http://arxiv.org/abs/1706.08612
Pith/arXiv arXiv 2017
-
[14]
V oxCeleb2: Deep Speaker Recognition,
J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep Speaker Recognition,” inInterspeech 2018. ISCA, Sep. 2018, pp. 1086–1090. [Online]. Available: http://www.isca-speech.org/archive/ Interspeech 2018/abstracts/1929.html
2018
-
[15]
Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,
Z. Zhang, L. Li, Y . Ding, and C. Fan, “Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3661–3670
2021
-
[16]
CelebV-HQ: A large-scale video facial attributes dataset,
H. Zhu, W. Wu, W. Zhu, L. Jiang, S. Tang, L. Zhang, Z. Liu, and C. C. Loy, “CelebV-HQ: A large-scale video facial attributes dataset,” inEuropean Conference on Computer Vision (ECCV), 2022, pp. 650– 667
2022
-
[17]
DH-FaceVid-1K: A large-scale high-quality dataset for face video generation,
D. Di, H. Feng, W. Sun, Y . Ma, H. Li, W. Chen, L. Fan, T. Su, and X. Yang, “DH-FaceVid-1K: A large-scale high-quality dataset for face video generation,” 2024
2024
-
[18]
MEAD: A large-scale audio-visual dataset for emotional talking-face generation,
K. Wang, Q. Wu, L. Song, Z. Yang, W. Wu, C. Qian, R. He, Y . Qiao, and C. C. Loy, “MEAD: A large-scale audio-visual dataset for emotional talking-face generation,” inEuropean Conference on Computer Vision (ECCV), 2020, pp. 700–717
2020
-
[19]
FaceForensics++: Learning to detect manipulated facial images,
A. R ¨ossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “FaceForensics++: Learning to detect manipulated facial images,” inIEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 1–11
2019
-
[20]
NTIRE 2019 Challenge on video super-resolution: Methods and results,
S. Nah, R. Timofte, S. Gu, S. Baik, S. Hong, G. Moon, S. Son, and K. Mu Lee, “NTIRE 2019 Challenge on video super-resolution: Methods and results,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0–0
2019
-
[21]
EDVR: Video restoration with enhanced deformable convolutional networks,
X. Wang, K. C. K. Chan, K. Yu, C. Dong, and C. C. Loy, “EDVR: Video restoration with enhanced deformable convolutional networks,” inIEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2019, pp. 1954–1963
2019
-
[22]
Basicvsr: The search for essential components in video super-resolution and beyond,
K. C. Chan, X. Wang, K. Yu, C. Dong, and C. C. Loy, “Basicvsr: The search for essential components in video super-resolution and beyond,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 4947–4956
2021
-
[23]
Basicvsr++: Improving video super-resolution with enhanced propagation and alignment,
K. C. Chan, S. Zhou, X. Xu, and C. C. Loy, “Basicvsr++: Improving video super-resolution with enhanced propagation and alignment,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 5972–5981
2022
-
[24]
Real-world video super- resolution: A benchmark dataset and a decomposition based learning scheme,
X. Yang, W. Xiang, H. Zeng, and L. Zhang, “Real-world video super- resolution: A benchmark dataset and a decomposition based learning scheme,” inIEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 4781–4790
2021
-
[25]
Investigating tradeoffs in real-world video super-resolution,
K. C. Chan, S. Zhou, X. Xu, and C. C. Loy, “Investigating tradeoffs in real-world video super-resolution,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5962–5971
2022
-
[26]
Generative face video coding techniques and standardization efforts: A review,
B. Chen, J. Wang, Y . Wang, Y . Ye, S. Wang, and Y . Li, “Generative face video coding techniques and standardization efforts: A review,” 2024
2024
-
[27]
Gemino: Practical and robust neural compression for video conferencing,
V . Sivaraman, S. Fouladi, S. Bhatt, S. Puffer, and K. Winstein, “Gemino: Practical and robust neural compression for video conferencing,” in USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2024. [23]�ITU-T Recommendation P.910�,Subjective video quality assessment methods for multimedia applications. International Telecommunication...
2024
-
[28]
Overview of the H.264 / A VC Video Coding Standard,
T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, “Overview of the H.264 / A VC Video Coding Standard,”IEEE Transactions On Circuits And Systems For Video Technology, p. 19, 2003
2003
-
[29]
Overview of the High Efficiency Video Coding (HEVC) Standard,
G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649– 1668, Dec. 2012, conference Name: IEEE Transactions on Circuits and Systems for Video Technology
2012
-
[30]
Overview of the Versatile Video Coding (VVC) Standard and its Applications,
B. Bross, Y .-K. Wang, Y . Ye, S. Liu, J. Chen, G. J. Sullivan, and J.- R. Ohm, “Overview of the Versatile Video Coding (VVC) Standard and its Applications,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736–3764, Oct. 2021, conference Name: IEEE Transactions on Circuits and Systems for Video Technology
2021
-
[31]
A Technical Overview of A V1,
J. Han, B. Li, D. Mukherjee, C.-H. Chiang, A. Grange, C. Chen, H. Su, S. Parker, S. Deng, U. Joshi, Y . Chen, Y . Wang, P. Wilkins, Y . Xu, and J. Bankoski, “A Technical Overview of A V1,”Proceedings of the IEEE, vol. 109, no. 9, pp. 1435–1462, Sep. 2021, conference Name: Proceedings of the IEEE
2021
-
[32]
FFV1 video coding format versions 0, 1, and 3,
M. Niedermayer, D. Rice, and J. Martinez, “FFV1 video coding format versions 0, 1, and 3,” IETF RFC 9043, 2022
2022
-
[33]
A crowdsourcing approach to video quality assessment,
B. Naderi and R. Cutler, “A crowdsourcing approach to video quality assessment,” inICASSP, 2024
2024
-
[34]
Crowdsourcing Quality of Experience Ex- periments,
S. Egger-Lampl, J. Redi, T. Hoßfeld, M. Hirth, S. M ¨oller, B. Naderi, C. Keimel, and D. Saupe, “Crowdsourcing Quality of Experience Ex- periments,” inQuality of Experience: Advanced Concepts, Applications and Methods. Springer, 2014, pp. 154–190
2014
-
[35]
CROWDMOS: An approach for crowdsourcing mean opinion score studies,
F. Ribeiro, D. Flor ˆencio, C. Zhang, and M. Seltzer, “CROWDMOS: An approach for crowdsourcing mean opinion score studies,” in2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2011, pp. 2416–2419, iSSN: 2379-190X
2011
-
[36]
The Ishihara test for color blindness
J. Clark, “The Ishihara test for color blindness.”American Journal of Physiological Optics, 1924
1924
-
[37]
An index of factorial simplicity,
H. F. Kaiser, “An index of factorial simplicity,”Psychometrika, vol. 39, no. 1, pp. 31–36, 1974
1974
-
[38]
Calculation of average PSNR differences between RD-curves,
G. Bjontegaard, “Calculation of average PSNR differences between RD-curves,” ITU-T Video Coding Experts Group (VCEG), Document VCEG-M33, 2001
2001
-
[39]
VMAF: The Journey Continues,
Z. Li, C. Bampis, J. Novak, A. Aaron, K. Swanson, A. Moorthy, and J. De Cock, “VMAF: The Journey Continues,” Tech. Rep., 2018. [Online]. Available: https://netflixtechblog.com/ vmaf-the-journey-continues-44b51ee9ed12
2018
-
[40]
WebRTC video processing and codec requirements,
H. T. Alvestrand, “WebRTC video processing and codec requirements,” RFC 7742, Internet Engineering Task Force, 2016. APPENDIXA DATACOLLECTIONDETAILS This section provides additional technical details on the recording pipeline summarized in Sec. 2.1. a) Pixel format selection.:Consumer webcams expose multiple output formats over the USB Video Class (UVC) p...
2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.