Pith. sign in

REVIEW 4 major objections 6 minor 42 references

A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read YouTube Shorts recommendations drift more sharply from politically sensitive seeds than from general topics, and keyframes are sufficient to measure the drift.

desk verdict The keyframe auditing pipeline is plausible, but the sensitive-vs-general comparison is confounded by seed topic breadth, so the headline drift claim does not hold as submitted. read the letter →

arxiv 2509.02543 v1 pith:P2RGNXLJ submitted 2025-09-02 cs.SI

classification cs.SI
keywords YouTubeShortsrecommendersystemauditingalgorithmicbiascontentdriftkeyframeextractionmultimodalembeddingsSouthChinaSeashort-formvideo
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small set of perceptually important frames, called keyframes, is enough to audit what a short-form video recommender does after a user watches a seed video. The authors collect 5,500 Shorts for the politically sensitive South China Sea topic and 5,500 for general YouTube categories, then generate captions from keyframes and embed both frames and captions in a shared space. They report that recommended Shorts drift away from the seed cluster in both visual and textual embeddings, and that the drift is considerably larger for South China Sea seeds than for general-category seeds. If the claim holds, keyframes give researchers and regulators a cheap, interpretable window into recommendation bias without decoding full videos.

What carries the argument

The load-bearing mechanism is a five-stage pipeline: PRISM selects a few perceptually salient frames from each Short; the Llama-3.2-11B-Vision-Instruct model writes a caption for each keyframe; CLIP ViT-G/14 embeds the frames and captions into one normalized semantic space; UMAP projects the embeddings for visual comparison; and structural metrics (variance, intra-cluster distance, Jensen-Shannon divergence, Wasserstein distance) quantify the seed-to-recommendation gap. The keyframe is the object that makes the audit tractable: roughly 30,000 keyframes stand in for 664,430 frames across 11,000 videos, preserving the visual content most likely to influence a viewer while removing the cost of full video processing.

What would settle it

Run the identical collection and embedding pipeline on a matched narrow, non-political topic—for example, 'traditional Japanese pottery' or 'vintage tractor restoration'—using seed keywords as tightly focused as the South China Sea set. If recommendation variance and intra-cluster distances rise to the South China Sea levels, the political-sensitivity explanation fails; if they stay near General Content levels, the explanation is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that keyframe-based content analysis reveals recommendation drift and potential filtering that metadata-based audits would miss. Starting from seed Shorts on the South China Sea and on general YouTube categories, the authors extract keyframes with PRISM, caption them with a vision-language model, and embed both modalities with CLIP. In their projections, recommended Shorts spread into a loose cloud around a tight South China Sea seed cluster, while general-category recommendations stay closer to their broader seed distribution. Quantitative divergence metrics—variance, intra-cluster distance, Jensen-Shannon divergence, and Wasserstein distance—are all higher for the sensitive topic, with frame embeddings diverging more sharply than caption embeddings. The conclusion is that keyframes serve as a scalable, interpretable proxy for detecting algorithmic bias in short-form video recommendations.

Load-bearing premise

The comparison in Section 3.1 uses a broad set of general YouTube categories as the control for a single narrow South China Sea topic, and the wider recommendation spread observed for South China Sea could occur for any narrow seed topic regardless of political sensitivity; if that is the case, the attribution of drift to political sensitivity collapses.

Editorial extensions

If this is right

  • Auditors can detect content drift in YouTube Shorts from keyframes alone, without decoding every frame of every video.
  • Visual framing is the stronger drift signal: frame embeddings separate the sensitive and general domains more sharply than caption embeddings do, so audits should weight visual features heavily.
  • A ten-step recommendation chain starting from a narrow sensitive topic moves into a wider, less coherent region of content space than a chain starting from a broad general category.
  • Because the caption and embedding models operate in zero-shot settings, the same pipeline can be pointed at new topics or languages without retraining the models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same pipeline could monitor drift for a topic over time, so a change in platform moderation or policy could be observed as a change in how far recommendations travel from the seed.
  • Because the data records each recommendation's depth in the chain, one could locate the depth at which divergence first spikes and distinguish early algorithmic exploration from deeper filtering.
  • The method transfers to other short-form surfaces such as TikTok or Instagram Reels, since it depends only on keyframes, captions, and embeddings rather than platform-internal data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes a keyframe-based audit methodology for YouTube Shorts recommendations. The authors collect 500 seed videos each for the South China Sea (SCS) topic and for general YouTube categories, scrape up to 10 recommended Shorts per seed (5,000 recommendations per dataset), extract keyframes with PRISM, generate captions with Llama-3.2-11B-Vision-Instruct, and embed frames and captions with CLIP. They then use UMAP projections, variance and intra-cluster distance statistics (Tables 6), and JSD/Wasserstein divergences (Table 7) to argue that SCS recommendations drift more strongly from their seeds than general-content recommendations, concluding that keyframes offer an interpretable lens for auditing algorithmic bias in short-form video.

Significance. The methodological contribution—keyframe-based multimodal drift measurement at scale—is potentially useful, and the pipeline is concrete enough to reproduce: no parameters are fitted to the outcome data, the data collection is described in detail, and the use of two modalities (visual keyframes and captions) is a genuine improvement over metadata-only audits. However, the paper's central empirical claim about politically sensitive topics relies on a confounded comparison and on point estimates without uncertainty quantification, so the substantive conclusion is not supported as submitted. The contribution would be significant if the confound were removed with a matched control and the results were accompanied by inferential statistics.

major comments (4)
  1. [Section 3.1/Table 1 and Table 6] The central comparison in RQ3 is confounded: the SCS condition is a single narrow geopolitical topic, while the General condition pools 15 broad YouTube categories. Table 6 shows seed variance of 4.29/4.23 for SCS versus 7.82/12.86 for General, so the larger recommendation variance for SCS (79.13/89.13) is predictable from seed breadth alone: a tight seed cluster leaves more room for expansion. The paper therefore has not shown that political sensitivity, rather than topic breadth, drives the drift difference; a matched narrow non-political control (e.g., a single specific hobby or technology topic) is required.
  2. [Section 5, Tables 6 and 7] All comparisons are point estimates with no confidence intervals, bootstrap, or significance tests, and each domain is represented by one keyword set, so the reported differences cannot be distinguished from sampling noise. The authors should report variability across seed subsets or bootstrap resamples and, minimally, a permutation test for the SCS-vs-General differences.
  3. [Section 5, Figures 2 and Tables 6-7] The drift metrics are computed on UMAP projections, but the UMAP hyperparameters, the normalization procedure for the 'normalized' scores, and the handling of UMAP stochasticity are not reported. Without these details the numerical values in Tables 6 and 7 are not reproducible and may be sensitive to projection artifacts; repeated runs with fixed random seeds and a sensitivity analysis are needed.
  4. [Section 6 and the Abstract] The conclusion moves from measured dispersion ('drift') to claims of 'bias' and 'filtering' (e.g., 'potential filtering' in the abstract), but the study provides no normative baseline or ground truth that would make larger dispersion interpretable as bias. At minimum the language should be hedged to 'divergence from seeds,' and the bias interpretation should be presented as a hypothesis rather than a finding.
minor comments (6)
  1. [General manuscript formatting] The manuscript retains ACM template placeholders (e.g., 'Conference acronym ’XX', 'Woodstock, NY', 'Received 20 February 2007', 'ACM Reference Format ... 2018'); these must be updated.
  2. [Table 6] Table 6 is captioned 'Normalized divergence scores' but the normalization operation is not defined; define it in the text or caption.
  3. [Figure 1] Figure 1 contains a stray label 'TriPSS' that does not appear to be explained; clarify or remove.
  4. [Section 3.3] Section 3.3 says the setup 'simulate[s] a neutral user environment,' but the exact browser configuration, bot-detection handling, and scraping time window are not described; this limits reproducibility of the recommendation collection.
  5. [References] The reference list includes authors' own unpublished or accepted work (e.g., [5]) and several non-archival preprints; the provenance of those claims should be checked at production.
  6. [Section 5, Figure 2] The visual claim that recommendations 'remain closer and more overlapping' for General content is not quantified in Figure 2; a quantitative overlap metric would support the narrative.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the drift metrics are empirical and not defined in terms of the paper's conclusions, and the self-citations are tools or background rather than load-bearing inputs.

full rationale

The paper's derivation chain is empirical and self-contained in the relevant sense. No equation in Sections 4-5 defines a reported metric in terms of the conclusion; the variance, intra-cluster distance, Jensen-Shannon divergence, and Wasserstein distance values are computed directly from CLIP and Llama-generated caption embeddings and are not fitted parameters. The self-citations to PRISM [5] and the authors' earlier bias audits [3,4,6,21-23] are tool and background citations: PRISM is benchmarked independently in Table 2, and the earlier audits do not supply the SCS-vs-General comparison reported here. The central claim that sensitive-topic recommendations drift more is an empirical observation, not a quantity forced by construction. The design concern that the SCS set is a single narrow topic while the General set is broad (Table 1) is a validity threat, since greater dispersion could be expected from seed breadth alone, but that is a confound in the experimental comparison, not a circular derivation; the measured dispersion statistics are not algebraically determined by seed variance in the paper. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central comparison relies on several domain assumptions about the proxies (keyframes, CLIP, captions) and on an unmatched control group. The paper adds no new entities; its main load-bearing choices are the keyframe extraction and the control group design, the latter being the most fragile. No parameters are fitted to data, but multiple thresholds and hyperparameters are chosen without sensitivity analysis.

free parameters (3)
  • Recommendation chain depth = 10
    Depth chosen without sensitivity analysis; deeper chains would likely increase measured drift, and the same depth may not be comparable across domains with different recommendation patterns.
  • UMAP hyperparameters = not reported
    UMAP requires n_neighbors and min_dist; these are not disclosed, and the visual clustering in Figure 2 depends on them.
  • PRISM keyframe selection thresholds = not reported
    PRISM's internal thresholds determine how many keyframes per video are extracted; the paper reports counts (15,361 vs 14,592) but no sensitivity analysis, and PRISM is the authors' own model.
assumptions (6)
  • domain assumption CLIP embeddings of keyframes and captions capture semantic content relevant to recommendation bias
    Invoked in Section 4.2; if CLIP is insensitive to the visual differences that matter for drift, the divergence metrics are ineffective.
  • domain assumption PRISM keyframes are representative of the visual content that drives recommendation dynamics
    Invoked in Section 3.4; PRISM is the authors' own keyframe extractor and its saliency definition may not align with what the recommendation algorithm uses.
  • domain assumption Captions generated by Llama-3.2-11B-Vision-Instruct faithfully describe keyframe content without systematic bias
    Invoked in Section 4.1; caption model biases would propagate into the caption embeddings and affect the drift comparison.
  • domain assumption Fresh logged-out Selenium sessions simulate a neutral user whose recommendation trajectory is informative
    Invoked in Section 3.3; logged-out recommendation behavior may differ from typical logged-in users, and the scrolling protocol is unspecified.
  • ad hoc to paper General Content keywords (YouTube category names) are a valid control for non-sensitive YouTube content
    Section 3.1; this is the paper's own choice, but the control is unmatched in breadth, which drives the main confound.
  • standard math Jensen-Shannon divergence and Wasserstein distance between embedding distributions measure meaningful recommendation drift
    Tables 6-7; these are standard distance measures, but their interpretation as drift requires that the embedding distribution reflects content topics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations." pith.science (2026). https://pith.science/paper/P2RGNXLJ

@misc{pith2026250902543,
  author       = {Pith},
  title        = {Pith review of: A Keyframe-Based Approach for Auditing Bias in YouTube Shorts Recommendations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P2RGNXLJ}},
  note         = {Machine review of arXiv:2509.02543}
}
read the original abstract

YouTube Shorts and other short-form video platforms now influence how billions engage with content, yet their recommendation systems remain largely opaque. Small shifts in promoted content can significantly impact user exposure, especially for politically sensitive topics. In this work, we propose a keyframe-based method to audit bias and drift in short-form video recommendations. Rather than analyzing full videos or relying on metadata, we extract perceptually salient keyframes, generate captions, and embed both into a shared content space. Using visual mapping across recommendation chains, we observe consistent shifts and clustering patterns that indicate topic drift and potential filtering. Comparing politically sensitive topics with general YouTube categories, we find notable differences in recommendation behavior. Our findings show that keyframes provide an efficient and interpretable lens for understanding bias in short-form video algorithms.

Figures

Figures reproduced from arXiv: 2509.02543 by the authors.

Figure 2
Figure 2. UMAP projections of keyframe and caption em [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 33 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac et al. 2022. Flamingo: a visual language model for few- shot learning. Advances in neural information processing systems , 35, 23716– 23736

  2. [2]

    Atlantic Council. 2025. Digital forensic research lab. https://www.atlanticcoun cil.org/programs/digital-forensic-research-lab/. Accessed: 2025-02-25. (2025)

  3. [3]

    Mert Can Cakmak and Nitin Agarwal. 2025. Unpacking algorithmic bias in youtube shorts by analyzing thumbnails. In Proceedings of the 58th Hawaii International Conference on System Sciences . (Jan. 2025). https://hdl.handle.net /10125/109144

  4. [4]

    Mert Can Cakmak, Nitin Agarwal, and Remi Oni. 2024. The bias beneath: analyzing drift in youtube’s algorithmic recommendations. Social Network Analysis and Mining, 14, 1, 171

  5. [5]

    Mert Can Cakmak, Nitin Agarwal, and Diwash Poudel. 2025. Prism: perceptual recognition for identifying standout moments in human-centric keyframe extraction. In Workshop Proceedings of the 19th International AAAI Conference on Web and Social Media . Accepted for presentation. Copenhagen, Denmark

  6. [6]

    Mert Can Cakmak, Obianuju Okeke, Ugochukwu Onyepunuka, Billy Spann, and Nitin Agarwal. 2023. Investigating bias in youtube recommendations: emo- tion, morality, and network dynamics in china-uyghur content. InInternational Conference on Complex Networks and Their Applications . Springer, 351–362

  7. [7]

    Centre for International Law, National University of Singapore. 2025. Ocean law and policy – south china sea. https://cil.nus.edu.sg/research/ocean-law-po licy/south-china-sea/. Accessed: 2025-02-25. (2025)

  8. [8]

    Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiang- nan He. 2023. Bias and debias in recommender system: a survey and future directions. ACM Transactions on Information Systems , 41, 3, 1–39

Show all 42 references
  1. [9]

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev

  2. [10]

    Yang Cong, Junsong Yuan, and Jiebo Luo. 2011. Towards scalable summariza- tion of consumer videos via sparse dictionary selection. IEEE Transactions on Multimedia, 14, 1, 66–75

  3. [11]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, 191–198

  4. [12]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: a large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

  5. [13]

    EntreResource. 2023. Youtube video categories – full list explained and which you should use in 2023. https://entreresource.com/youtube-video-categories-f ull-list-explained-and-which-you-should-use/. Accessed: 2025-02-25. (2023)

  6. [14]

    Rajesh Garapati and Manomita Chakraborty. 2025. Recommender systems in the digital age: a comprehensive review of methods, challenges, and applica- tions. Knowledge and Information Systems , 1–45

  7. [15]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  8. [16]

    Georg Groh, Stefan Birnkammerer, and Valeria Köllhofer. 2012. Social recom- mender systems. Recommender systems for the social web , 3–42

  9. [17]

    Hussam Habib and Rishab Nithyanand. 2025. Youtube recommendations rein- force negative emotions: auditing algorithmic bias with emotionally-agentic sock puppets. arXiv preprint arXiv:2501.15048

  10. [18]

    Muhammad Haroon, Anshuman Chhabra, Xin Liu, Prasant Mohapatra, Zubair Shafiq, and Magdalena Wojcieszak. 2022. Youtube, the great radicalizer? au- diting and mitigating ideological biases in youtube recommendations. arXiv preprint arXiv:2203.10666

  11. [19]

    Hazem Ibrahim, HyunSeok Daniel Jang, Nouar Aldahoul, Aaron R Kaufman, Talal Rahwan, and Yasir Zaki. 2025. Tiktok’s recommendations skewed to- wards republican content during the 2024 us presidential race. arXiv preprint arXiv:2501.17831

  12. [20]

    Chao Jia et al. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning. PMLR, 4904–4916

  13. [21]

    Baris Kirdemir and Nitin Agarwal. 2021. Exploring bias and information bubbles in youtube’s video recommendation networks. In International Conference on Complex Networks and Their Applications . Springer, 166–177

  14. [22]

    Baris Kirdemir, Joseph Kready, Esther Mead, Muhammad Nihal Hussain, and Nitin Agarwal. 2021. Examining video recommendation bias on youtube. In International Workshop on Algorithmic Bias in Search and Recommendation . Springer, 106–116

  15. [23]

    Baris Kirdemir, Joseph Kready, Esther Mead, Muhammad Nihal Hussain, Nitin Agarwal, and Donald Adjeroh. 2021. Assessing bias in youtube’s video rec- ommendation algorithm in a cross-lingual and cross-topical context. In Social, Cultural, and Behavioral Modeling: 14th Internatio...

  16. [24]

    Hugo Laurençon et al. 2023. Introducing idefics: an open reproduction of state- of-the-art visual language model. https://huggingface.co/blog/idefics. Hugging Face. (2023)

  17. [25]

    Mark Ledwich and Anna Zaitsev. 2019. Algorithmic extremism: examining youtube’s rabbit hole of radicalization. arXiv preprint arXiv:1912.11211

  18. [26]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888– 12900

  19. [27]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proce...

  20. [28]

    Xiao Lin, Xiaokai Chen, Linfeng Song, Jingwei Liu, Biao Li, and Peng Jiang. 2023. Tree based progressive regression model for watch-time prediction in short- video recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 4497–4506

  21. [29]

    Masoud Mansoury, Himan Abdollahpouri, Mykola Pechenizkiy, Bamshad Mobasher, and Robin Burke. 2020. Feedback loop and bias amplification in recommender systems. In Proceedings of the 29th ACM international conference on information & knowledge management , 2145–2148

  22. [30]

    Leland McInnes, John Healy, and James Melville. 2018. Umap: uniform man- ifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426

  23. [31]

    Meta. 2025. Meta-llama/llama-3.2-11b-vision-instruct. Hugging Face. Accessed: January 4, 2025. (2025). https://huggingface.co/meta-llama/Llama-3.2-11B-Vis ion-Instruct

  24. [32]

    Alec Radford et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning . PmLR, 8748–8763. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Cakmak & Agarwal

  25. [33]

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8317–8326

  26. [34]

    Streamers. 2024. Youtube scraper. Accessed: 2024-01-10. APIFY. https://apify.c om/streamers/youtube-scraper

  27. [35]

    Mitchell Wortsman. 2023. Reaching 80% zero-shot accuracy with openclip: vit-g/14. https://laion.ai/blog/giant-openclip/. (2023)

  28. [36]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: new similarity metrics for semantic inference over event descriptions. Transactions of the association for computa- tional linguistics, 2, 67–78

  29. [37]

    Ruohan Zhan, Changhua Pei, Qiang Su, Jianfeng Wen, Xueliang Wang, Guanyu Mu, Dong Zheng, Peng Jiang, and Kun Gai. 2022. Deconfounding duration bias in watch-time prediction for video recommendation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and da...

  30. [38]

    Bin Zhao and Eric P Xing. 2014. Quasi real-time summarization for consumer videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2513–2520

  31. [39]

    Yu Zheng, Chen Gao, Jingtao Ding, Lingling Yi, Depeng Jin, Yong Li, and Meng Wang. 2022. Dvr: micro-video recommendation optimizing watch-time-gain under duration bias. In Proceedings of the 30th ACM International Conference on Multimedia, 334–345

  32. [40]

    Wengang Zhou, Weichao Zhao, Hezhen Hu, Zecheng Li, and Houqiang Li. 2024. Scaling up multimodal pre-training for sign language understanding. arXiv preprint arXiv:2408.08544. Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009

  33. [2017]

    In Proceedings of the IEEE conference on computer vision and pattern recognition , 6904–6913

    Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , 6904–6913

  34. [2023]

    In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion

    Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion. Also see the open_clip codebase at https://github.com/mlfoundations/ope n_clip, 2818–2829

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.