REVIEW 4 major objections 6 minor 42 references
A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read YouTube Shorts recommendations drift more sharply from politically sensitive seeds than from general topics, and keyframes are sufficient to measure the drift.
desk verdict The keyframe auditing pipeline is plausible, but the sensitive-vs-general comparison is confounded by seed topic breadth, so the headline drift claim does not hold as submitted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a five-stage pipeline: PRISM selects a few perceptually salient frames from each Short; the Llama-3.2-11B-Vision-Instruct model writes a caption for each keyframe; CLIP ViT-G/14 embeds the frames and captions into one normalized semantic space; UMAP projects the embeddings for visual comparison; and structural metrics (variance, intra-cluster distance, Jensen-Shannon divergence, Wasserstein distance) quantify the seed-to-recommendation gap. The keyframe is the object that makes the audit tractable: roughly 30,000 keyframes stand in for 664,430 frames across 11,000 videos, preserving the visual content most likely to influence a viewer while removing the cost of full video processing.
What would settle it
Run the identical collection and embedding pipeline on a matched narrow, non-political topic—for example, 'traditional Japanese pottery' or 'vintage tractor restoration'—using seed keywords as tightly focused as the South China Sea set. If recommendation variance and intra-cluster distances rise to the South China Sea levels, the political-sensitivity explanation fails; if they stay near General Content levels, the explanation is supported.
Extended reading notes
Core claim
The paper's central claim is that keyframe-based content analysis reveals recommendation drift and potential filtering that metadata-based audits would miss. Starting from seed Shorts on the South China Sea and on general YouTube categories, the authors extract keyframes with PRISM, caption them with a vision-language model, and embed both modalities with CLIP. In their projections, recommended Shorts spread into a loose cloud around a tight South China Sea seed cluster, while general-category recommendations stay closer to their broader seed distribution. Quantitative divergence metrics—variance, intra-cluster distance, Jensen-Shannon divergence, and Wasserstein distance—are all higher for the sensitive topic, with frame embeddings diverging more sharply than caption embeddings. The conclusion is that keyframes serve as a scalable, interpretable proxy for detecting algorithmic bias in short-form video recommendations.
Load-bearing premise
The comparison in Section 3.1 uses a broad set of general YouTube categories as the control for a single narrow South China Sea topic, and the wider recommendation spread observed for South China Sea could occur for any narrow seed topic regardless of political sensitivity; if that is the case, the attribution of drift to political sensitivity collapses.
Editorial extensions
If this is right
- Auditors can detect content drift in YouTube Shorts from keyframes alone, without decoding every frame of every video.
- Visual framing is the stronger drift signal: frame embeddings separate the sensitive and general domains more sharply than caption embeddings do, so audits should weight visual features heavily.
- A ten-step recommendation chain starting from a narrow sensitive topic moves into a wider, less coherent region of content space than a chain starting from a broad general category.
- Because the caption and embedding models operate in zero-shot settings, the same pipeline can be pointed at new topics or languages without retraining the models.
Reading between the lines
- The same pipeline could monitor drift for a topic over time, so a change in platform moderation or policy could be observed as a change in how far recommendations travel from the seed.
- Because the data records each recommendation's depth in the chain, one could locate the depth at which divergence first spikes and distinguish early algorithmic exploration from deeper filtering.
- The method transfers to other short-form surfaces such as TikTok or Instagram Reels, since it depends only on keyframes, captions, and embeddings rather than platform-internal data.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a keyframe-based audit methodology for YouTube Shorts recommendations. The authors collect 500 seed videos each for the South China Sea (SCS) topic and for general YouTube categories, scrape up to 10 recommended Shorts per seed (5,000 recommendations per dataset), extract keyframes with PRISM, generate captions with Llama-3.2-11B-Vision-Instruct, and embed frames and captions with CLIP. They then use UMAP projections, variance and intra-cluster distance statistics (Tables 6), and JSD/Wasserstein divergences (Table 7) to argue that SCS recommendations drift more strongly from their seeds than general-content recommendations, concluding that keyframes offer an interpretable lens for auditing algorithmic bias in short-form video.
Significance. The methodological contribution—keyframe-based multimodal drift measurement at scale—is potentially useful, and the pipeline is concrete enough to reproduce: no parameters are fitted to the outcome data, the data collection is described in detail, and the use of two modalities (visual keyframes and captions) is a genuine improvement over metadata-only audits. However, the paper's central empirical claim about politically sensitive topics relies on a confounded comparison and on point estimates without uncertainty quantification, so the substantive conclusion is not supported as submitted. The contribution would be significant if the confound were removed with a matched control and the results were accompanied by inferential statistics.
major comments (4)
- [Section 3.1/Table 1 and Table 6] The central comparison in RQ3 is confounded: the SCS condition is a single narrow geopolitical topic, while the General condition pools 15 broad YouTube categories. Table 6 shows seed variance of 4.29/4.23 for SCS versus 7.82/12.86 for General, so the larger recommendation variance for SCS (79.13/89.13) is predictable from seed breadth alone: a tight seed cluster leaves more room for expansion. The paper therefore has not shown that political sensitivity, rather than topic breadth, drives the drift difference; a matched narrow non-political control (e.g., a single specific hobby or technology topic) is required.
- [Section 5, Tables 6 and 7] All comparisons are point estimates with no confidence intervals, bootstrap, or significance tests, and each domain is represented by one keyword set, so the reported differences cannot be distinguished from sampling noise. The authors should report variability across seed subsets or bootstrap resamples and, minimally, a permutation test for the SCS-vs-General differences.
- [Section 5, Figures 2 and Tables 6-7] The drift metrics are computed on UMAP projections, but the UMAP hyperparameters, the normalization procedure for the 'normalized' scores, and the handling of UMAP stochasticity are not reported. Without these details the numerical values in Tables 6 and 7 are not reproducible and may be sensitive to projection artifacts; repeated runs with fixed random seeds and a sensitivity analysis are needed.
- [Section 6 and the Abstract] The conclusion moves from measured dispersion ('drift') to claims of 'bias' and 'filtering' (e.g., 'potential filtering' in the abstract), but the study provides no normative baseline or ground truth that would make larger dispersion interpretable as bias. At minimum the language should be hedged to 'divergence from seeds,' and the bias interpretation should be presented as a hypothesis rather than a finding.
minor comments (6)
- [General manuscript formatting] The manuscript retains ACM template placeholders (e.g., 'Conference acronym ’XX', 'Woodstock, NY', 'Received 20 February 2007', 'ACM Reference Format ... 2018'); these must be updated.
- [Table 6] Table 6 is captioned 'Normalized divergence scores' but the normalization operation is not defined; define it in the text or caption.
- [Figure 1] Figure 1 contains a stray label 'TriPSS' that does not appear to be explained; clarify or remove.
- [Section 3.3] Section 3.3 says the setup 'simulate[s] a neutral user environment,' but the exact browser configuration, bot-detection handling, and scraping time window are not described; this limits reproducibility of the recommendation collection.
- [References] The reference list includes authors' own unpublished or accepted work (e.g., [5]) and several non-archival preprints; the provenance of those claims should be checked at production.
- [Section 5, Figure 2] The visual claim that recommendations 'remain closer and more overlapping' for General content is not quantified in Figure 2; a quantitative overlap metric would support the narrative.
Circularity Check
No circular derivation found; the drift metrics are empirical and not defined in terms of the paper's conclusions, and the self-citations are tools or background rather than load-bearing inputs.
full rationale
The paper's derivation chain is empirical and self-contained in the relevant sense. No equation in Sections 4-5 defines a reported metric in terms of the conclusion; the variance, intra-cluster distance, Jensen-Shannon divergence, and Wasserstein distance values are computed directly from CLIP and Llama-generated caption embeddings and are not fitted parameters. The self-citations to PRISM [5] and the authors' earlier bias audits [3,4,6,21-23] are tool and background citations: PRISM is benchmarked independently in Table 2, and the earlier audits do not supply the SCS-vs-General comparison reported here. The central claim that sensitive-topic recommendations drift more is an empirical observation, not a quantity forced by construction. The design concern that the SCS set is a single narrow topic while the General set is broad (Table 1) is a validity threat, since greater dispersion could be expected from seed breadth alone, but that is a confound in the experimental comparison, not a circular derivation; the measured dispersion statistics are not algebraically determined by seed variance in the paper. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Recommendation chain depth =
10
- UMAP hyperparameters =
not reported
- PRISM keyframe selection thresholds =
not reported
assumptions (6)
- domain assumption CLIP embeddings of keyframes and captions capture semantic content relevant to recommendation bias
- domain assumption PRISM keyframes are representative of the visual content that drives recommendation dynamics
- domain assumption Captions generated by Llama-3.2-11B-Vision-Instruct faithfully describe keyframe content without systematic bias
- domain assumption Fresh logged-out Selenium sessions simulate a neutral user whose recommendation trajectory is informative
- ad hoc to paper General Content keywords (YouTube category names) are a valid control for non-sensitive YouTube content
- standard math Jensen-Shannon divergence and Wasserstein distance between embedding distributions measure meaningful recommendation drift
Cite this review
Pith. "Pith review of A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations." pith.science (2026). https://pith.science/paper/P2RGNXLJ
@misc{pith2026250902543,
author = {Pith},
title = {Pith review of: A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations},
year = {2026},
howpublished = {\url{https://pith.science/paper/P2RGNXLJ}},
note = {Machine review of arXiv:2509.02543}
}
read the original abstract
YouTube Shorts and other short-form video platforms now influence how billions engage with content, yet their recommendation systems remain largely opaque. Small shifts in promoted content can significantly impact user exposure, especially for politically sensitive topics. In this work, we propose a keyframe-based method to audit bias and drift in short-form video recommendations. Rather than analyzing full videos or relying on metadata, we extract perceptually salient keyframes, generate captions, and embed both into a shared content space. Using visual mapping across recommendation chains, we observe consistent shifts and clustering patterns that indicate topic drift and potential filtering. Comparing politically sensitive topics with general YouTube categories, we find notable differences in recommendation behavior. Our findings show that keyframes provide an efficient and interpretable lens for understanding bias in short-form video algorithms.
Figures
Reference graph
Works this paper leans on
-
[1]
Jean-Baptiste Alayrac et al. 2022. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems , 35, 23716– 23736
work page 2022
-
[2]
Atlantic Council. 2025. Digital forensic research lab. https://www.atlanticcoun cil.org/programs/digital-forensic-research-lab/. Accessed: 2025-02-25. (2025)
work page 2025
-
[3]
Mert Can Cakmak and Nitin Agarwal. 2025. Unpacking algorithmic bias in youtube shorts by analyzing thumbnails. In Proceedings of the 58th Hawaii International Conference on System Sciences . (Jan. 2025). https://hdl.handle.net /10125/109144
work page 2025
-
[4]
Mert Can Cakmak, Nitin Agarwal, and Remi Oni. 2024. The bias beneath: analyzing drift in youtube’s algorithmic recommendations. Social Network Analysis and Mining, 14, 1, 171
work page 2024
-
[5]
Mert Can Cakmak, Nitin Agarwal, and Diwash Poudel. 2025. Prism: perceptual recognition for identifying standout moments in human-centric keyframe extraction. In Workshop Proceedings of the 19th International AAAI Conference on Web and Social Media . Accepted for presentation. Copenhagen, Denmark
work page 2025
-
[6]
Mert Can Cakmak, Obianuju Okeke, Ugochukwu Onyepunuka, Billy Spann, and Nitin Agarwal. 2023. Investigating bias in youtube recommendations: emo- tion, morality, and network dynamics in china-uyghur content. InInternational Conference on Complex Networks and Their Applications . Springer, 351–362
work page 2023
-
[7]
Centre for International Law, National University of Singapore. 2025. Ocean law and policy – south china sea. https://cil.nus.edu.sg/research/ocean-law-po licy/south-china-sea/. Accessed: 2025-02-25. (2025)
work page 2025
-
[8]
Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiang- nan He. 2023. Bias and debias in recommender system: a survey and future directions. ACM Transactions on Information Systems , 41, 3, 1–39
2023
Show all 42 references
-
[9]
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev
-
[10]
Yang Cong, Junsong Yuan, and Jiebo Luo. 2011. Towards scalable summariza- tion of consumer videos via sparse dictionary selection. IEEE Transactions on Multimedia, 14, 1, 66–75
2011
-
[11]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, 191–198
2016
-
[12]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255
2009
-
[13]
EntreResource. 2023. Youtube video categories – full list explained and which you should use in 2023. https://entreresource.com/youtube-video-categories-f ull-list-explained-and-which-you-should-use/. Accessed: 2025-02-25. (2023)
2023
-
[14]
Rajesh Garapati and Manomita Chakraborty. 2025. Recommender systems in the digital age: a comprehensive review of methods, challenges, and applica- tions. Knowledge and Information Systems , 1–45
2025
-
[15]
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh
-
[16]
Georg Groh, Stefan Birnkammerer, and Valeria Köllhofer. 2012. Social recom- mender systems. Recommender systems for the social web , 3–42
2012
-
[17]
Hussam Habib and Rishab Nithyanand. 2025. Youtube recommendations rein- force negative emotions: auditing algorithmic bias with emotionally-agentic sock puppets. arXiv preprint arXiv:2501.15048
2025 arXiv
-
[18]
Muhammad Haroon, Anshuman Chhabra, Xin Liu, Prasant Mohapatra, Zubair Shafiq, and Magdalena Wojcieszak. 2022. Youtube, the great radicalizer? au- diting and mitigating ideological biases in youtube recommendations. arXiv preprint arXiv:2203.10666
2022 arXiv
-
[19]
Hazem Ibrahim, HyunSeok Daniel Jang, Nouar Aldahoul, Aaron R Kaufman, Talal Rahwan, and Yasir Zaki. 2025. Tiktok’s recommendations skewed to- wards republican content during the 2024 us presidential race. arXiv preprint arXiv:2501.17831
2025 arXiv
-
[20]
Chao Jia et al. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning. PMLR, 4904–4916
2021
-
[21]
Baris Kirdemir and Nitin Agarwal. 2021. Exploring bias and information bubbles in youtube’s video recommendation networks. In International Conference on Complex Networks and Their Applications . Springer, 166–177
2021
-
[22]
Baris Kirdemir, Joseph Kready, Esther Mead, Muhammad Nihal Hussain, and Nitin Agarwal. 2021. Examining video recommendation bias on youtube. In International Workshop on Algorithmic Bias in Search and Recommendation . Springer, 106–116
2021
-
[23]
Baris Kirdemir, Joseph Kready, Esther Mead, Muhammad Nihal Hussain, Nitin Agarwal, and Donald Adjeroh. 2021. Assessing bias in youtube’s video rec- ommendation algorithm in a cross-lingual and cross-topical context. In Social, Cultural, and Behavioral Modeling: 14th Internatio...
2021
-
[24]
Hugo Laurençon et al. 2023. Introducing idefics: an open reproduction of state- of-the-art visual language model. https://huggingface.co/blog/idefics. Hugging Face. (2023)
2023
-
[25]
Mark Ledwich and Anna Zaitsev. 2019. Algorithmic extremism: examining youtube’s rabbit hole of radicalization. arXiv preprint arXiv:1912.11211
2019 arXiv
-
[26]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888– 12900
2022
-
[27]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...
2014
-
[28]
Xiao Lin, Xiaokai Chen, Linfeng Song, Jingwei Liu, Biao Li, and Peng Jiang. 2023. Tree based progressive regression model for watch-time prediction in short- video recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 4497–4506
2023
-
[29]
Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management , 2145–2148
2020
-
[30]
Leland McInnes, John Healy, and James Melville. 2018. Umap: uniform man- ifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426
2018 arXiv
-
[31]
Meta. 2025. Meta-llama/llama-3.2-11b-vision-instruct. Hugging Face. Accessed: January 4, 2025. (2025). https://huggingface.co/meta-llama/Llama-3.2-11B-Vis ion-Instruct
2025
-
[32]
Alec Radford et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning . PmLR, 8748–8763. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Cakmak & Agarwal
2021
-
[33]
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8317–8326
2019
-
[34]
Streamers. 2024. Youtube scraper. Accessed: 2024-01-10. APIFY. https://apify.c om/streamers/youtube-scraper
2024
-
[35]
Mitchell Wortsman. 2023. Reaching 80% zero-shot accuracy with openclip: vit-g/14. https://laion.ai/blog/giant-openclip/. (2023)
2023
-
[36]
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Transactions of the association for computa- tional linguistics, 2, 67–78
2014
-
[37]
Ruohan Zhan, Changhua Pei, Qiang Su, Jianfeng Wen, Xueliang Wang, Guanyu Mu, Dong Zheng, Peng Jiang, and Kun Gai. 2022. Deconfounding duration bias in watch-time prediction for video recommendation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and da...
2022
-
[38]
Bin Zhao and Eric P Xing. 2014. Quasi real-time summarization for consumer videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2513–2520
2014
-
[39]
Yu Zheng, Chen Gao, Jingtao Ding, Lingling Yi, Depeng Jin, Yong Li, and Meng Wang. 2022. Dvr: micro-video recommendation optimizing watch-time-gain under duration bias. In Proceedings of the 30th ACM International Conference on Multimedia, 334–345
2022
-
[40]
Wengang Zhou, Weichao Zhao, Hezhen Hu, Zecheng Li, and Houqiang Li. 2024. Scaling up multimodal pre-training for sign language understanding. arXiv preprint arXiv:2408.08544. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
2024 arXiv
-
[2017]
In Proceedings of the IEEE conference on computer vision and pattern recognition , 6904–6913
Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , 6904–6913
-
[2023]
In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion
Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion. Also see the open_clip codebase at https://github.com/mlfoundations/ope n_clip, 2818–2829
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.