REVIEW 3 major objections 5 minor 28 references
On YouTube Search API Use in Research
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Running identical YouTube search queries over twelve weeks shows the API returns shifting historical video sets that track topical popularity around the query date.
desk verdict Solid audit of YouTube Search API instability; the core variability result holds, but the popularity-randomization mechanism and 'impossible to sample' claims overreach the data, which the paper's own totalResults numbers undercut. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the repeated identical-query audit design: for each of six topics, the same keyword query was sent for every hour of a 28-day historical window, and the whole 28-day sweep was repeated every five days for twelve weeks, producing 4,032 queries per sweep. The key quantities are Jaccard similarity, the size of the intersection of two video-ID sets divided by the size of their union, between successive sets $J(S_t, S_{t-1})$ and between the current and first sets $J(S_t, S_1)$, which quantify decay over time. A second-order Markov chain over the presence or absence of each video in successive collections reveals the rolling-window sampling dynamics, showing that a video is more likely to stay present or absent when its two previous states matched. The SHAP decomposition of a Poisson-objective gradient-boosting model identifies which video and channel features predict how often a video is returned, pointing to views, duration, and topic-pool size as the strongest predictors.
What would settle it
Pick a historical hour with a small, known candidate pool, say ten videos, query it on many dates spanning high- and low-interest periods for the topic, and compare the returned sets: if the sets vary with query-date interest, the paper's mechanism is supported; if they are identical whenever the pool is below the API's per-page cap, the central claim is refuted.
Extended reading notes
Core claim
The central claim is that the Search: list endpoint randomizes its results over time even for fully historical queries, making representative historical sampling effectively impossible, especially during non-peak topical periods. Concretely, Jaccard similarity between the video-ID set from the first collection and a later collection decays over the twelve weeks, and videos appear at time $t$ that were absent at $t-1$ even though the query window lies entirely in the past. The paper rules out ceiling effects: per-hour returned counts stay far below the 50-per-page maximum, and the correlation between hourly returned volume and set similarity is weakly positive, not negative. It interprets this as evidence that the API samples from an empirical distribution of topical interest, sometimes returning zero videos when the relative density of interest is low. A second-order Markov chain over presence and absence in successive collections shows videos drop in and out in a rolling-window fashion, and a SHAP analysis of a gradient-boosting model indicates that view count, shorter duration, and smaller topic pools are associated with more consistent returns.
Load-bearing premise
The argument assumes that the API's publishedAfter and publishedBefore parameters genuinely limit the search to the requested one-hour window rather than returning videos sampled from a whole day and only then filtering them by the hour bounds.
Editorial extensions
If this is right
- A researcher who queries the same historical topic on two different dates may end up with two different video sets for the same day, so longitudinal datasets built from search snapshots inherit the query dates as a hidden variable.
- Time-splitting a query into hourly bins does not defeat the API's sampling: per-hour returned counts stay far below the per-page maximum, and similarity does not drop where more videos are returned.
- Smaller query pools produce more stable results, so breaking a topic into narrower sub-queries or adding AND terms is a more promising strategy than splitting the time window.
- ID-based endpoints such as Video:list and CommentThreads:list show stable behavior, making channel-level discovery via Channels:list plus PlaylistItems:list a viable alternative when consistency matters.
- The Markov-chain analysis implies videos enter and leave the returned set in rolling windows, so the set a researcher sees on a given day is a transient window over the API's internal distribution, not a cumulative archive.
Reading between the lines
- If the API's sampling tracks query-period topical interest, then any dataset assembled from repeated search calls encodes the researcher's observation calendar as much as the historical event itself; longitudinal analyses should therefore model query date as a confound.
- The stable ID-based endpoints provide a natural ground-truth check the paper does not run: randomly sample channels relevant to a topic, pull complete uploads via Channels:list and PlaylistItems:list, and compare that universe to search results to quantify how much of the historical record search misses.
- The view-count and duration effects could be partly an artifact of the popularity-sampling mechanism rather than independent ranking criteria: shorter, popular videos may simply be overrepresented in the empirical interest distribution that the API appears to draw from.
- A testable next step would be to run the same audit on a topic with a deliberately engineered upload schedule, uploading a known set of videos at known times, to recover the API's sampling weights directly.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an observational audit of the YouTube Data API Search:list endpoint, running identical historical queries for six topics (BLM, Brexit, Capitol Riots, Grammys, Higgs Boson, World Cup) every five days over roughly twelve weeks. For each collection, the author issued hourly time-binned queries across a 28-day window around each topic's focal date, then measured Jaccard similarity of returned video ID sets across collections. The main descriptive finding is that returned sets diverge substantially over time, even for fully historical queries, and that this divergence is not explained by content deletion or by the behavior of ID-based endpoints such as Videos:list and CommentThreads:list. The paper interprets this variability as evidence that the API randomly samples from topic-wide empirical popularity distributions, withholds videos during low-activity periods, and tends to return shorter, more popular videos. It concludes with practical recommendations for researchers, including query narrowing and use of ID-based endpoints.
Significance. If the descriptive variability result holds, the paper is a useful and timely contribution for the measurement community: it quantifies a replicability threat for pipeline research built on the YouTube Search API, provides an ID-endpoint comparison that strengthens the finding, and ships code and data. The controls for content deletion and the comparison against stable ID-based endpoints are genuine strengths, as is the relatively long observation window of twelve weeks. However, the paper's mechanism-level claims—that the API randomizes on the basis of relative topical popularity and that representative non-peak sampling is nearly impossible—are load-bearing for the abstract and are not entailed by the data as analyzed. The descriptive Jaccard-decay result is solid; the mechanism interpretation needs substantial reframing or additional validation, which is why I recommend major revision.
major comments (3)
- [§1 Abstract, §4.2] The central mechanism claim in the abstract—that the API randomizes returned videos based on the relative popularity of the topic and withholds data during low-activity periods—rests on the assumption that the hourly publishedAfter/publishedBefore bounds produce true hourly cohorts. The paper itself undermines this assumption. Section 3 states that the API documentation makes no commitment to ordering beyond daily granularity and that 'order may take precedence over time-filtering'; Section 5 then reports in Table 3 that pageInfo.totalResults has a modal value of 1,000,000 for three topics even though every query is scoped to a single hour, and explicitly notes that 'time constraints in queries do not affect the total pool of available videos in the API's results set' and 'may only filter the response after results are returned.' If the hourly bounds are applied after sampling from a day-level pool, then the per-hour frequency distributions in Table 2 and Figure 2 are not estimates of hourly upload activity, the 'zero-video' hours do not demonstrate deliberate withholding, and the 'empirical distribution of topical interest' explanation in Section 4.2 loses its foundation. The Jaccard-decay finding survives, but the mechanism claim does not. The paper should either restrict its abstract and Section 4.2 to the descriptive variability finding, or add a test that can distinguish hour-level sampling from day-level sampling followed by post-filtering (for example, by comparing queries whose totalResults is below the 500-video ceiling against those at the ceiling).
- [Abstract and §4.2] The claim that the API 'randomizes' returned videos based on relative popularity is not directly supported by the measurements. The study observes variability in returned sets, but variability alone does not establish randomization: it could result from a deterministic but time-dependent ranking, from undocumented recency biases, or from backend caching changes. The paper's own evidence—particularly the unusually high modal totalResults values—is consistent with the API constructing a large candidate pool and then ordering or filtering within that pool, but the 'relative popularity' mechanism is inferred rather than tested. To make this claim load-bearing, the authors would need a more direct manipulation, such as comparing queries with artificially narrowed candidate pools (e.g., by topic or region parameters) or a temporal analysis of when specific videos enter and leave the result set. As written, the abstract's 'randomize based on the relative popularity' phrase overstates what the data show; I recommend softening it to describe time-dependent variability and treating the popularity-based mechanism as a hypothesis, not a finding.
- [§5, Tables 5–7] The claim that the API 'may prioritize shorter, more popular videos' is presented as a main result but is only weakly supported by the predictive models: the gradient boosting model reports R^2 = 0.19 on the test set, the binned ordinal regression has pseudo-R^2 = 0.079, and the non-binned ordinal regression has pseudo-R^2 = 0.04. These fit values indicate that the features explain a small fraction of variance in return frequency. The directional consistency across the robustness checks is reassuring, but the paper should be careful not to imply that video popularity and duration are strong determinants of API behavior. I would ask the authors to state explicitly in Section 5 that the model explains less than 20% of the variance and that the identified features are weak predictors, which would align the prose with the reported statistics.
minor comments (5)
- [Figure 1] The 'error bars' in Figure 1 are actually set differences (the complement of the intersection relative to each set), not uncertainty intervals. Please relabel them in the caption or the text to avoid confusion.
- [Table 2 caption] The caption says 'N is the number of videos retained after all hours with no videos returned across collections are dropped,' but N appears to denote the number of retained hourly bins, not videos. Please correct this wording.
- [Section 3] The sentence beginning 'On such days, it can alternatively be the case that order may take precedence over time-filtering' is important for the paper's argument, but it is not connected to the later totalResults analysis. Consider moving or signposting this caveat when Table 3 is discussed, so that the reader can evaluate both pieces of evidence together.
- [Appendix B] In the JSON snippet for the general parameters, publishedBefore is listed, but the topic-specific parameter list only mentions publishedAfter. Please add the publishedBefore date range for each topic for completeness.
- [Section 5] There is a typo: 'the numberof videos to be returned' should read 'the number of videos to be returned.'
Circularity Check
No significant circularity: the audit's descriptive claims are self-contained, and the hourly-cohort caveat is a validity limitation rather than a circular reduction.
full rationale
The paper is an observational audit of the YouTube Search API. Its central findings—temporal variability in returned video sets, weak correlations between hourly volume and Jaccard similarity, Markov-chain persistence of video presence, and feature associations with return frequency—are all measured directly from query responses. No parameter is fitted to a target outcome and then renamed as a prediction; the LightGBM/SHAP analysis is explicitly explanatory with a modest reported fit (R2 = 0.19), not a claim that the model's output validates the model's inputs. The paper's interpretation that the API samples from an empirical popularity distribution is an inference from observed patterns, not an assumption built into the measurement. The most serious concern raised by the reader—that hourly publishedAfter/publishedBefore bounds may not produce true hourly cohorts—is a real identification threat to the mechanism claim, and the paper itself flags this in Section 3 ('the YouTube API's documentation makes no commitment to order results beyond a daily granularity') and in Section 5 via the pageInfo.totalResults observation. However, that is a question of internal validity or generalizability, not circularity: the descriptive variability findings do not depend on the contested mechanism, and no equation or definition reduces the conclusion to an input. There is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in by citation. The external citation to Covington et al. is used only as an analogy for how the API might behave. Accordingly, the derivation chain is self-contained and the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The YouTube Data API's publishedAfter and publishedBefore parameters filter results at hourly granularity.
- domain assumption The six chosen topics and single API client are representative of YouTube Search API behavior for research use.
- domain assumption pageInfo.totalResults in the search response reflects the size of the potential video pool.
Cite this review
Pith. "Pith review of On YouTube Search API Use in Research." pith.science (2026). https://pith.science/paper/NSSJPV5R
@misc{pith2026250604422,
author = {Pith},
title = {Pith review of: On YouTube Search API Use in Research},
year = {2026},
howpublished = {\url{https://pith.science/paper/NSSJPV5R}},
note = {Machine review of arXiv:2506.04422}
}
read the original abstract
YouTube is among the most widely-used platforms worldwide, and has seen a lot of recent academic attention. Despite its popularity and the number of studies conducted on it, much less is understood about the way in which YouTube's Data API, and especially the Search endpoint, operates. In this paper, we analyze the API's behavior by running identical queries across a period of 12 weeks. Our findings show that the search endpoint returns highly variable results between queries. Specifically, the API seems to randomize returned videos based on the relative popularity of the respective topic during the query period, making it nearly impossible to obtain representative historical video samples, especially during non-peak topical periods. Our results also suggest that the API may prioritize shorter, more popular videos, although the role of channel popularity is not as clear. We conclude with suggested strategies for researchers using the API for data collection, as well as future research directions on expanding the API's use-cases.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2022. Deradicalizing YouTube: Characterization, Detection, and Personalization of Religiously Intolerant Arabic Videos.Proceedings of the ACM on Human-Computer Interaction6, CSCW2 (Nov. 2022), 1–25. doi:10.1145/3555618
doi:10.1145/3555618 2022
-
[2]
Thales Bertaglia, Catalina Goanta, and Adriana Iamnitchi. 2024. The Monetisa- tion of Toxicity: Analysing YouTube Content Creators and Controversy-Driven Engagement. InProceedings of the 4th International Workshop on Open Challenges in Online Social Networks (OASIS ’24). Association for Computing Machinery, New York, NY, USA, 1–9. doi:10.1145/3677117.3685005
-
[3]
Frederick Choi, Charlotte Lambert, Vinay Koshy, Sowmya Pratipati, Tue Do, and Eshwar Chandrasekharan. 2025. Creator Hearts: Investigating the Impact Positive Signals from YouTube Creators in Shaping Comment Section Behavior. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–18. doi:10.1145/3706598.3713521
arXiv 2025
-
[4]
Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. InProceedings of the 10th ACM Conference on Recommender Systems. ACM, Boston Massachusetts USA, 191–198. doi:10.1145/ 2959100.2959190
arXiv 2016
-
[5]
Myrsini Gkolemi, Panagiotis Papadopoulos, Evangelos Markatos, and Nicolas Kourtellis. 2022. YouTubers Not madeForKids: Detecting Channels Sharing Inappropriate Videos Targeting Children. In14th ACM Web Science Conference
work page 2022
-
[7]
Eslam Hussein, Prerna Juneja, and Tanushree Mitra. 2020. Measuring Misin- formation in Video Search Platforms: An Audit Study on YouTube.Proceed- ings of the ACM on Human-Computer Interaction4, CSCW1 (May 2020), 1–27. doi:10.1145/3392854
doi:10.1145/3392854 2020
-
[8]
Hayoung Jung, Prerna Juneja, and Tanushree Mitra. 2025. Algorithmic Behaviors Across Regions: A Geolocation Audit of YouTube Search for COVID-19 Misin- formation Between the United States and South Africa. doi:10.48550/arXiv.2409. 10168 arXiv:2409.10168 [cs]
-
[9]
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. InAdvances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/hash/ 6449f44a102fde848669bdd9eb6b76fa-Abstract.html
work page 2017
Show all 28 references
-
[10]
Emaan Bilal Khan, Nida Tanveer, Aima Shahid, Mohammad Jaffer Iqbal, Haashim Ali Mirza, Armish Javed, Ihsan Ayyub Qazi, and Zafar Ayyub Qazi
-
[11]
Voelker, and Sarah Meiklejohn
Enze Liu, George Kappos, Eric Mugnier, Luca Invernizzi, Stefan Savage, David Tao, Kurt Thomas, Geoffrey M. Voelker, and Sarah Meiklejohn. 2024. Give and Take: An End-To-End Investigation of Giveaway Scam Conversion Rates. In Proceedings of the 2024 ACM on Internet Measurement ...
2024
-
[12]
Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpret- ing Model Predictions. InAdvances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/hash/ 8a20a8621978632d76c43dfd28b67767-Abstract.html
2017
-
[13]
Robin Mamié, Manoel Horta Ribeiro, and Robert West. 2021. Are Anti-Feminist Communities Gateways to the Far Right? Evidence from Reddit and YouTube. In Proceedings of the 13th ACM Web Science Conference 2021 (WebSci ’21). Association for Computing Machinery, New York, NY, USA,...
2021 doi
-
[14]
Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: automated black-box testing of RESTful web APIs. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Virtual Denmark, 682–685. doi:10.1145/3460319.3469082
2021
-
[15]
Lloyd May, Keita Ohshiro, Khang Dang, Sripathi Sridhar, Jhanvi Pai, Magdalena Fuentes, Sooyeon Lee, and Mark Cartwright. 2024. Unspoken Sound: Identifying Trends in Non-Speech Audio Captioning on YouTube. InProceedings of the CHI Conference on Human Factors in Computing System...
2024
-
[16]
Ryan McGrady, Kevin Zheng, Rebecca Curran, Jason Baumgartner, and Ethan Zuckerman. 2023. Dialing for Videos: A Random Sample of YouTube.Journal of Quantitative Description: Digital Media3 (Dec. 2023). doi:10.51685/jqd.2023.022
2023 doi
-
[17]
I Searched for a Religious Song in Amharic and Got Sexual Content Instead
Hellina Hailu Nigatu and Inioluwa Deborah Raji. 2024. “I Searched for a Religious Song in Amharic and Got Sexual Content Instead”: Investigating Online Harm in Low-Resourced Languages on YouTube.. InThe 2024 ACM Conference on Fairness, Accountability, and Transparency. ACM, Ri...
2024
-
[18]
Kostantinos Papadamou, Antonis Papasavva, Savvas Zannettou, Jeremy Black- burn, Nicolas Kourtellis, Ilias Leontiadis, Gianluca Stringhini, and Michael Sirivianos. 2020. Disturbed YouTube for Kids: Characterizing and Detect- ing Inappropriate Videos Targeting Young Children.Pro...
2020 doi
-
[19]
It Is Just a Flu
Kostantinos Papadamou, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Michael Sirivianos. 2022. “It Is Just a Flu”: Assessing the Effect of Watch History on YouTube’s Pseudoscientific Video Recommendations.Proceedings of the International ...
2022 doi
-
[20]
How over is On YouTube Search API Use in Research IMC ’25, October 28–31, 2025, Madison, WI, USA it?
Kostantinos Papadamou, Savvas Zannettou, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Michael Sirivianos. 2021. "How over is On YouTube Search API Use in Research IMC ’25, October 28–31, 2025, Madison, WI, USA it?" Understanding the Incel Community on You...
2021 doi
-
[22]
Annamaria Porreca, Francesca Scozzari, and Marta Di Nicola. 2020. Using text mining and sentiment analysis to analyse YouTube Italian videos concerning vaccination.BMC Public Health20, 1 (Feb. 2020), 259. doi:10.1186/s12889-020- 8342-4
2020 doi
-
[23]
Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio A. F. Almeida, and Wagner Meira. 2020. Auditing radicalization pathways on YouTube. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* ’20). Association for Computing Machinery, ...
2020
-
[24]
Bernhard Rieder, Ariadna Matamoros-Fernández, and Òscar Coromina. 2018. From ranking algorithms to ‘ranking cultures’: Investigating the modulation of visibility in YouTube search results.Convergence24, 1 (Feb. 2018), 50–68. doi:10.1177/1354856517736982 Publisher: SAGE Publica...
2018 doi
-
[25]
Bernhard Rieder, Adrian Padilla, and Oscar Coromina. 2025. Forgetful by Design? A Critical Audit of YouTube’s Search API for Academic Research. doi:10.48550/ arXiv.2506.11727 arXiv:2506.11727 [cs]
2025 doi
-
[26]
Daniel Röchert, Muriel Weitzel, and Björn Ross. 2020. The homogeneity of right- wing populist and radical content in YouTube recommendations. InInternational Conference on Social Media and Society. ACM, Toronto ON Canada, 245–254. doi:10.1145/3400806.3400835
2020
-
[27]
Caroline Violot, Tuğrulcan Elmas, Igor Bilogrevic, and Mathias Humbert. 2024. Shorts vs. Regular Videos on YouTube: A Comparative Analysis of User En- gagement and Content Creation Trends. InACM Web Science Conference. ACM, Stuttgart Germany, 213–223. doi:10.1145/3614419.3644023
2024
-
[28]
part": "snippet
Leon Yin and Megan Brown. 2018. SMAPPNYU/youtube-data-api. doi:10.5281/ ZENODO.1414418 A Ethics This work makes sole use of video, comment, or channel IDs and high-level metadata (number of views, likes, etc.) as data points and does not analyze content beyond this point. The ...
2018
-
[2022]
doi:10.1145/3501247.3531556
ACM, Barcelona Spain, 370–381. doi:10.1145/3501247.3531556
-
[2024]
InProceedings of the ACM Web Conference 2024
Analyzing Ad Exposure and Content in Child-Oriented Videos on YouTube. InProceedings of the ACM Web Conference 2024. ACM, Singapore Singapore, 1215–1226. doi:10.1145/3589334.3645585
2024
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.