{"id":"cb4b7bc5-8b78-4e72-91e7-a2bd5a5e6eda","arxiv_id":"2509.02543","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"This study finds that YouTube Shorts recommendations diverge more sharply from seed content for South China Sea queries than for general YouTube categories, using a new keyframe-based visual and textual analysis.","lead":"This paper proposes a keyframe-based method to audit bias and drift in YouTube Shorts recommendations, using still frames and captions to compare recommended videos to seed videos. If validated, it would offer a scalable way to monitor how short-video algorithms steer users on sensitive topics, though the current comparison is confounded.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Topic-breadth confound invalidates the sensitive-vs-general comparison: one narrow SCS seed set is compared against 15 broad categories, so the Table 6 dispersion differences would be expected from seed breadth alone. A matched narrow non-political control is needed to support the sensitivity claim.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern, and I agree with it. The confound is not a stylistic quibble: because the dependent variables are dispersion measures and the seed distributions differ by construction, the headline finding is predicted by a null model with no sensitivity effect. A broad seed set already spans the recommendation space, so recommendations from it have less room to expand; a narrow seed set starts from a tight cluster, so even a fixed exploration kernel produces larger dispersion increases. Thus the comparison in Table 6 cannot distinguish 'politically sensitive topic' from 'narrow topic'. The paper provides no matched narrow non-political control, no statistical uncertainty quantification, and no permutation or bootstrap test, so the claimed sensitivity-specific effect is not established. The proposed re-run is feasible with the existing scraping pipeline: only the seed topics need to be added, and the same metrics can be recomputed. If the concern lands, the method contribution may still be a useful auditing tool, but the paper's empirical conclusion about politically sensitive topics must be revised or re-framed as a hypothesis. Therefore I keep the reader's verdict unchanged.","tokens_in":8539,"tokens_out":4010,"duration_ms":40317,"concrete_test":"Collect 5-10 non-political narrow seed topics matched to SCS's seed variance (e.g., 'aquascaping', 'speedcubing tutorials', 'vintage motorcycle restoration'), each with 500 seed Shorts and 10-level recommendation chains using the same APIFY and Selenium pipeline as Sections 3.2-3.3. Recompute Table 6 metrics (seed/rec variance and intra-cluster distances) for each topic with bootstrap 95% CIs. If narrow non-political topics yield rec variance and intra-distance comparable to SCS (around 79/89 caption/frame), the political-sensitivity attribution fails; if they remain near General levels, the confound is resolved and the claim gains support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 6) attributes higher drift and dispersion for South China Sea recommendations to political sensitivity. The experimental contrast, however, varies two factors at once: topic sensitivity and seed-topic breadth. The SCS seed set is a single narrow geopolitical topic (Table 1), whereas the General dataset consists of 500 seeds spread across 15 broad YouTube categories. Table 6 shows this imbalance directly: seed variance for General is 7.82 (caption) and 12.86 (frame), while SCS seed variance is 4.29 and 4.23. The outcome measures are dispersion statistics; a tight seed cluster can mechanically produce larger relative expansion of the recommendation set even if the recommendation algorithm behaves identically. Any narrow non-political topic would be expected to show similar exploration, yet no such matched control is included. Additionally, Tables 6 and 7 report point estimates with no confidence intervals, bootstrap, or significance tests, so even the raw difference is not established as robust. Because the entire sensitive-vs-general conclusion rests on this unmatched comparison, the central empirical claim is unsupported as submitted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a keyframe-based audit methodology for YouTube Shorts recommendations. The authors collect 500 seed videos each for the South China Sea (SCS) topic and for general YouTube categories, scrape up to 10 recommended Shorts per seed (5,000 recommendations per dataset), extract keyframes with PRISM, generate captions with Llama-3.2-11B-Vision-Instruct, and embed frames and captions with CLIP. They then use UMAP projections, variance and intra-cluster distance statistics (Tables 6), and JSD/Wasserstein divergences (Table 7) to argue that SCS recommendations drift more strongly from their seeds than general-content recommendations, concluding that keyframes offer an interpretable lens for auditing algorithmic bias in short-form video.","tokens_in":8685,"tokens_out":4995,"duration_ms":44352,"significance":"The methodological contribution—keyframe-based multimodal drift measurement at scale—is potentially useful, and the pipeline is concrete enough to reproduce: no parameters are fitted to the outcome data, the data collection is described in detail, and the use of two modalities (visual keyframes and captions) is a genuine improvement over metadata-only audits. However, the paper's central empirical claim about politically sensitive topics relies on a confounded comparison and on point estimates without uncertainty quantification, so the substantive conclusion is not supported as submitted. The contribution would be significant if the confound were removed with a matched control and the results were accompanied by inferential statistics.","major_comments":[{"comment":"The central comparison in RQ3 is confounded: the SCS condition is a single narrow geopolitical topic, while the General condition pools 15 broad YouTube categories. Table 6 shows seed variance of 4.29/4.23 for SCS versus 7.82/12.86 for General, so the larger recommendation variance for SCS (79.13/89.13) is predictable from seed breadth alone: a tight seed cluster leaves more room for expansion. The paper therefore has not shown that political sensitivity, rather than topic breadth, drives the drift difference; a matched narrow non-political control (e.g., a single specific hobby or technology topic) is required.","section":"Section 3.1/Table 1 and Table 6"},{"comment":"All comparisons are point estimates with no confidence intervals, bootstrap, or significance tests, and each domain is represented by one keyword set, so the reported differences cannot be distinguished from sampling noise. The authors should report variability across seed subsets or bootstrap resamples and, minimally, a permutation test for the SCS-vs-General differences.","section":"Section 5, Tables 6 and 7"},{"comment":"The drift metrics are computed on UMAP projections, but the UMAP hyperparameters, the normalization procedure for the 'normalized' scores, and the handling of UMAP stochasticity are not reported. Without these details the numerical values in Tables 6 and 7 are not reproducible and may be sensitive to projection artifacts; repeated runs with fixed random seeds and a sensitivity analysis are needed.","section":"Section 5, Figures 2 and Tables 6-7"},{"comment":"The conclusion moves from measured dispersion ('drift') to claims of 'bias' and 'filtering' (e.g., 'potential filtering' in the abstract), but the study provides no normative baseline or ground truth that would make larger dispersion interpretable as bias. At minimum the language should be hedged to 'divergence from seeds,' and the bias interpretation should be presented as a hypothesis rather than a finding.","section":"Section 6 and the Abstract"}],"minor_comments":[{"comment":"The manuscript retains ACM template placeholders (e.g., 'Conference acronym ’XX', 'Woodstock, NY', 'Received 20 February 2007', 'ACM Reference Format ... 2018'); these must be updated.","section":"General manuscript formatting"},{"comment":"Table 6 is captioned 'Normalized divergence scores' but the normalization operation is not defined; define it in the text or caption.","section":"Table 6"},{"comment":"Figure 1 contains a stray label 'TriPSS' that does not appear to be explained; clarify or remove.","section":"Figure 1"},{"comment":"Section 3.3 says the setup 'simulate[s] a neutral user environment,' but the exact browser configuration, bot-detection handling, and scraping time window are not described; this limits reproducibility of the recommendation collection.","section":"Section 3.3"},{"comment":"The reference list includes authors' own unpublished or accepted work (e.g., [5]) and several non-archival preprints; the provenance of those claims should be checked at production.","section":"References"},{"comment":"The visual claim that recommendations 'remain closer and more overlapping' for General content is not quantified in Figure 2; a quantitative overlap metric would support the narrative.","section":"Section 5, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The confound in the main comparison is severe; even after a matched control is added, the lack of inferential statistics would need to be remedied. I would advise the editor that acceptance in the current form is not defensible, but the proposed method has enough merit that a substantial revision with new data collection could be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the keyframe pipeline is a plausible way to audit short-form recommendation drift, and the authors put in real data-collection effort. But the central comparison that is supposed to show political sensitivity is confounded by seed topic breadth, and that confound is load-bearing. As submitted, the headline claim does not hold.\n\nWhat is actually new: combining PRISM keyframes, CLIP embeddings, and Llama captions into an auditing pipeline for YouTube Shorts is a genuine extension of prior thumbnail-based work. The internal consistency check—caption embeddings and frame embeddings show parallel patterns—is a nice touch. Collecting 5,500 videos per domain and staying in a fresh browser session to approximate a neutral user is more careful than some auditing work. The motivation for using keyframes instead of full video is practical and well stated.\n\nThe soft spots are real and central. The General Content control spans 15 broad YouTube categories, while SCS is one narrow geopolitical topic. Table 6 shows the consequence directly: seed variance is 7.82/12.86 for General vs 4.29/4.23 for SCS. A recommendation system that explores outward from a tight seed cluster will mechanically produce larger relative dispersion even if its behavior is identical. Without a matched narrow non-political topic, the drift difference cannot be attributed to political sensitivity. That is not a minor caveat; it is the paper’s main empirical claim.\n\nThe paper also reports no confidence intervals or significance tests, so even the raw differences are not established as robust. There is no code or data release, which makes independent verification difficult. PRISM is the authors’ own extractor and the paper gives no validation that its keyframes are better than random frames or thumbnails for this task. That matters because a keyframe pipeline that is not validated as a proxy for content could inherit its own biases.\n\nNone of this kills the method. The confound is fixable: add one or two narrow non-political topics (e.g., a specific hobby or sport) with the same seed count, run non-parametric tests, and release artifacts. The paper is a useful starting point for auditing short-form video, and the topic is important enough that a serious referee should engage with it. My recommendation: send it to peer review, but the revision needs to address the matched-control issue and the missing statistics before any empirical claim can be credited.","headline":"The keyframe auditing pipeline is plausible, but the sensitive-vs-general comparison is confounded by seed topic breadth, so the headline drift claim does not hold as submitted.","tokens_in":9298,"tokens_out":2701,"would_cite":false,"duration_ms":25104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"YouTube Shorts recommendations drift more sharply from politically sensitive seeds than from general topics, and keyframes are sufficient to measure the drift.","keywords":["YouTube Shorts","recommender system auditing","algorithmic bias","content drift","keyframe extraction","multimodal embeddings","South China Sea","short-form video"],"falsifier":"Run the identical collection and embedding pipeline on a matched narrow, non-political topic—for example, 'traditional Japanese pottery' or 'vintage tractor restoration'—using seed keywords as tightly focused as the South China Sea set. If recommendation variance and intra-cluster distances rise to the South China Sea levels, the political-sensitivity explanation fails; if they stay near General Content levels, the explanation is supported.","tokens_in":8232,"feed_emoji":"🎬","tokens_out":11859,"duration_ms":95018,"temperature":0.7,"pith_summary":"This paper tries to establish that a small set of perceptually important frames, called keyframes, is enough to audit what a short-form video recommender does after a user watches a seed video. The authors collect 5,500 Shorts for the politically sensitive South China Sea topic and 5,500 for general YouTube categories, then generate captions from keyframes and embed both frames and captions in a shared space. They report that recommended Shorts drift away from the seed cluster in both visual and textual embeddings, and that the drift is considerably larger for South China Sea seeds than for general-category seeds. If the claim holds, keyframes give researchers and regulators a cheap, interpretable window into recommendation bias without decoding full videos.","feed_headline":"Keyframes show Shorts drift far from sensitive-topic seeds","feed_subtitle":"A viewer starting on a contested topic is pulled into content far from the original seed.","key_machinery":"The load-bearing mechanism is a five-stage pipeline: PRISM selects a few perceptually salient frames from each Short; the Llama-3.2-11B-Vision-Instruct model writes a caption for each keyframe; CLIP ViT-G/14 embeds the frames and captions into one normalized semantic space; UMAP projects the embeddings for visual comparison; and structural metrics (variance, intra-cluster distance, Jensen-Shannon divergence, Wasserstein distance) quantify the seed-to-recommendation gap. The keyframe is the object that makes the audit tractable: roughly 30,000 keyframes stand in for 664,430 frames across 11,000 videos, preserving the visual content most likely to influence a viewer while removing the cost of full video processing.","core_discovery":"The paper's central claim is that keyframe-based content analysis reveals recommendation drift and potential filtering that metadata-based audits would miss. Starting from seed Shorts on the South China Sea and on general YouTube categories, the authors extract keyframes with PRISM, caption them with a vision-language model, and embed both modalities with CLIP. In their projections, recommended Shorts spread into a loose cloud around a tight South China Sea seed cluster, while general-category recommendations stay closer to their broader seed distribution. Quantitative divergence metrics—variance, intra-cluster distance, Jensen-Shannon divergence, and Wasserstein distance—are all higher for the sensitive topic, with frame embeddings diverging more sharply than caption embeddings. The conclusion is that keyframes serve as a scalable, interpretable proxy for detecting algorithmic bias in short-form video recommendations.","pith_inferences":["The same pipeline could monitor drift for a topic over time, so a change in platform moderation or policy could be observed as a change in how far recommendations travel from the seed.","Because the data records each recommendation's depth in the chain, one could locate the depth at which divergence first spikes and distinguish early algorithmic exploration from deeper filtering.","The method transfers to other short-form surfaces such as TikTok or Instagram Reels, since it depends only on keyframes, captions, and embeddings rather than platform-internal data."],"forward_implications":["Auditors can detect content drift in YouTube Shorts from keyframes alone, without decoding every frame of every video.","Visual framing is the stronger drift signal: frame embeddings separate the sensitive and general domains more sharply than caption embeddings do, so audits should weight visual features heavily.","A ten-step recommendation chain starting from a narrow sensitive topic moves into a wider, less coherent region of content space than a chain starting from a broad general category.","Because the caption and embedding models operate in zero-shot settings, the same pipeline can be pointed at new topics or languages without retraining the models."],"supporting_citations":[{"why":"It supplies the PRISM keyframe-extraction method that reduces each Short to perceptually salient frames.","marker":"[5]"},{"why":"It generates the keyframe captions whose text embeddings form the semantic half of the shared space.","marker":"[31]"},{"why":"It provides the CLIP vision-language embeddings used to project frames and captions into a common space.","marker":"[32]"},{"why":"It identifies the specific OpenCLIP ViT-G/14 model whose zero-shot performance the paper benchmarks.","marker":"[35]"},{"why":"It supplies UMAP, the dimensionality-reduction method used to visualize seed and recommendation clusters.","marker":"[30]"},{"why":"It is the scraper used to collect Shorts video IDs, since the official API does not expose Shorts.","marker":"[34]"},{"why":"It is the earlier thumbnail-level audit of YouTube Shorts bias that the keyframe method extends.","marker":"[3]"},{"why":"It documents sentiment drift for sensitive China-related recommendations, motivating the South China Sea case.","marker":"[4]"},{"why":"It defines YouTube's content taxonomy used to select the General Content baseline.","marker":"[13]"}],"fun_headline_variants":["Keyframes expose Shorts bias beyond metadata","New keyframe audit tracks YouTube Shorts drift","Visual keyframes reveal Shorts recommendation bias","Keyframe lens spots topic drift in Shorts","Keyframe audit shows Shorts filter sensitive topics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison in Section 3.1 uses a broad set of general YouTube categories as the control for a single narrow South China Sea topic, and the wider recommendation spread observed for South China Sea could occur for any narrow seed topic regardless of political sensitivity; if that is the case, the attribution of drift to political sensitivity collapses.","fun_headline_variants_meta":{"raw":{"variants":["Keyframes expose Shorts bias beyond metadata","New keyframe audit tracks YouTube Shorts drift","Visual keyframes reveal Shorts recommendation bias","Keyframe lens spots topic drift in Shorts","Keyframe audit shows Shorts filter sensitive topics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2272,"prompt_tokens":828,"completion_tokens":1444,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":1375}},"tokens_in":444,"tokens_out":1444,"duration_ms":8430,"temperature":1.0,"reasoning_tokens":1375,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:36:58.184517+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical collection and embedding pipeline on a matched narrow, non-political topic—for example, 'traditional Japanese pottery' or 'vintage tractor restoration'—using seed keywords as tightly focused as the South China Sea set. If recommendation variance and intra-cluster distances rise to the South China Sea levels, the political-sensitivity explanation fails; if they stay near General Content levels, the explanation is supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the PRISM keyframe-extraction method that reduces each Short to perceptually salient frames."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the CLIP vision-language embeddings used to project frames and captions into a common space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It identifies the specific OpenCLIP ViT-G/14 model whose zero-shot performance the paper benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the scraper used to collect Shorts video IDs, since the official API does not expose Shorts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the earlier thumbnail-level audit of YouTube Shorts bias that the keyframe method extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It documents sentiment drift for sensitive China-related recommendations, motivating the South China Sea case."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines YouTube's content taxonomy used to select the General Content baseline."}],"review_version":2}