{"id":"9246fe56-490c-4a25-8400-1c259c3252ef","arxiv_id":"2506.04422","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"YouTube's Search API returns substantially different video sets for the same historical query across collection dates, making time-sliced keyword sampling unreliable for building representative datasets.","lead":"Running identical historical queries through YouTube's Search API 16 times over 12 weeks produced video sets that overlap progressively less with each passing collection. This matters because many YouTube research datasets are built from this API, so the instability undermines reproducibility and the representativeness of historical samples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing assumption behind the paper's randomization mechanism is that hourly publishedAfter/Before bounds produce true hourly cohorts; the paper's own totalResults observation indicates the API ignores time bounds when constructing its result pool, so the 'non-peak withholding'…","rationale":"The reader's weakest_assumption and my concern coincide: the paper's mechanistic interpretation rests on the unverified assumption that publishedAfter/publishedBefore produce hour-precise cohorts. The paper itself provides evidence against this assumption in Section 3 and Table 3, yet the abstract states the popularity-based randomization and the near-impossibility of representative historical sampling as if established. This is a genuine soft spot because it targets the central claim's mechanism, not merely a secondary detail. My analysis does not change the reader's verdict: the descriptive finding of temporal variability is credible and worth reporting, but the paper should be conditional on either validating the hourly filtering or tempering the mechanism and impossibility claims. The concrete test I propose directly checks the hourly-cohort assumption using ground truth and the paper's own metadata. I agree with the reader's identification of this as the weakest link, and I see no other concern that is more load-bearing. The paper deserves credit for transparently noting the limitation and for providing code and data; with a simple validation step, the mechanism claim could be either supported or appropriately weakened.","tokens_in":13053,"tokens_out":6810,"duration_ms":68551,"concrete_test":"To settle whether the hourly cohorts are real, run the same hourly search query (identical publishedAfter/publishedBefore) 10–20 times across different dates for a topic with a known ground-truth set of matching videos in that hour, where the ground truth is established independently (e.g., by enumerating a seed channel's uploads via the PlaylistItems endpoint, or by taking the union of many search queries and cross-checking timestamps). If the API consistently returns the complete known set, the filter is honored; if it returns varying strict subsets, sampling occurs at or below the hourly level. Additionally, as a self-check, verify that every returned video's actual snippet.publishedAt falls within the requested hour—if any fall outside, the time filter is leaky.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim in the abstract—that the API randomizes based on topical popularity and makes non-peak periods nearly impossible to sample—depends on the per-hour frequency distributions in Table 2 and Figure 2 reflecting genuine hourly cohorts. Section 3 concedes that the API's documentation promises at most daily ordering granularity and that order may take precedence over time filtering, so returned videos may be sampled from a whole day and only later filtered to the requested hour. Table 3 strengthens this suspicion: pageInfo.totalResults is reported with a modal value of 1,000,000 for three topics even though every query is scoped to a single hour, implying the API computes its candidate pool without applying the time bounds. If this is correct, the large number of zero-video hours and the relative scarcity at non-peak times are consequences of a day-level sample of up to 500 videos, not of a deliberate withholding of inactive periods. The Jaccard decay in Figure 1 remains a valid, important finding, but the 'empirical distribution of topical interest' explanation in Section 4.2 and the impossibility claim in the abstract are not entailed by the data. The paper itself flags this limitation, which is why the concern does not invalidate the core descriptive result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an observational audit of the YouTube Data API Search:list endpoint, running identical historical queries for six topics (BLM, Brexit, Capitol Riots, Grammys, Higgs Boson, World Cup) every five days over roughly twelve weeks. For each collection, the author issued hourly time-binned queries across a 28-day window around each topic's focal date, then measured Jaccard similarity of returned video ID sets across collections. The main descriptive finding is that returned sets diverge substantially over time, even for fully historical queries, and that this divergence is not explained by content deletion or by the behavior of ID-based endpoints such as Videos:list and CommentThreads:list. The paper interprets this variability as evidence that the API randomly samples from topic-wide empirical popularity distributions, withholds videos during low-activity periods, and tends to return shorter, more popular videos. It concludes with practical recommendations for researchers, including query narrowing and use of ID-based endpoints.","tokens_in":13219,"tokens_out":3708,"duration_ms":38434,"significance":"If the descriptive variability result holds, the paper is a useful and timely contribution for the measurement community: it quantifies a replicability threat for pipeline research built on the YouTube Search API, provides an ID-endpoint comparison that strengthens the finding, and ships code and data. The controls for content deletion and the comparison against stable ID-based endpoints are genuine strengths, as is the relatively long observation window of twelve weeks. However, the paper's mechanism-level claims—that the API randomizes on the basis of relative topical popularity and that representative non-peak sampling is nearly impossible—are load-bearing for the abstract and are not entailed by the data as analyzed. The descriptive Jaccard-decay result is solid; the mechanism interpretation needs substantial reframing or additional validation, which is why I recommend major revision.","major_comments":[{"comment":"The central mechanism claim in the abstract—that the API randomizes returned videos based on the relative popularity of the topic and withholds data during low-activity periods—rests on the assumption that the hourly publishedAfter/publishedBefore bounds produce true hourly cohorts. The paper itself undermines this assumption. Section 3 states that the API documentation makes no commitment to ordering beyond daily granularity and that 'order may take precedence over time-filtering'; Section 5 then reports in Table 3 that pageInfo.totalResults has a modal value of 1,000,000 for three topics even though every query is scoped to a single hour, and explicitly notes that 'time constraints in queries do not affect the total pool of available videos in the API's results set' and 'may only filter the response after results are returned.' If the hourly bounds are applied after sampling from a day-level pool, then the per-hour frequency distributions in Table 2 and Figure 2 are not estimates of hourly upload activity, the 'zero-video' hours do not demonstrate deliberate withholding, and the 'empirical distribution of topical interest' explanation in Section 4.2 loses its foundation. The Jaccard-decay finding survives, but the mechanism claim does not. The paper should either restrict its abstract and Section 4.2 to the descriptive variability finding, or add a test that can distinguish hour-level sampling from day-level sampling followed by post-filtering (for example, by comparing queries whose totalResults is below the 500-video ceiling against those at the ceiling).","section":"§1 Abstract, §4.2"},{"comment":"The claim that the API 'randomizes' returned videos based on relative popularity is not directly supported by the measurements. The study observes variability in returned sets, but variability alone does not establish randomization: it could result from a deterministic but time-dependent ranking, from undocumented recency biases, or from backend caching changes. The paper's own evidence—particularly the unusually high modal totalResults values—is consistent with the API constructing a large candidate pool and then ordering or filtering within that pool, but the 'relative popularity' mechanism is inferred rather than tested. To make this claim load-bearing, the authors would need a more direct manipulation, such as comparing queries with artificially narrowed candidate pools (e.g., by topic or region parameters) or a temporal analysis of when specific videos enter and leave the result set. As written, the abstract's 'randomize based on the relative popularity' phrase overstates what the data show; I recommend softening it to describe time-dependent variability and treating the popularity-based mechanism as a hypothesis, not a finding.","section":"Abstract and §4.2"},{"comment":"The claim that the API 'may prioritize shorter, more popular videos' is presented as a main result but is only weakly supported by the predictive models: the gradient boosting model reports R^2 = 0.19 on the test set, the binned ordinal regression has pseudo-R^2 = 0.079, and the non-binned ordinal regression has pseudo-R^2 = 0.04. These fit values indicate that the features explain a small fraction of variance in return frequency. The directional consistency across the robustness checks is reassuring, but the paper should be careful not to imply that video popularity and duration are strong determinants of API behavior. I would ask the authors to state explicitly in Section 5 that the model explains less than 20% of the variance and that the identified features are weak predictors, which would align the prose with the reported statistics.","section":"§5, Tables 5–7"}],"minor_comments":[{"comment":"The 'error bars' in Figure 1 are actually set differences (the complement of the intersection relative to each set), not uncertainty intervals. Please relabel them in the caption or the text to avoid confusion.","section":"Figure 1"},{"comment":"The caption says 'N is the number of videos retained after all hours with no videos returned across collections are dropped,' but N appears to denote the number of retained hourly bins, not videos. Please correct this wording.","section":"Table 2 caption"},{"comment":"The sentence beginning 'On such days, it can alternatively be the case that order may take precedence over time-filtering' is important for the paper's argument, but it is not connected to the later totalResults analysis. Consider moving or signposting this caveat when Table 3 is discussed, so that the reader can evaluate both pieces of evidence together.","section":"Section 3"},{"comment":"In the JSON snippet for the general parameters, publishedBefore is listed, but the topic-specific parameter list only mentions publishedAfter. Please add the publishedBefore date range for each topic for completeness.","section":"Appendix B"},{"comment":"There is a typo: 'the numberof videos to be returned' should read 'the number of videos to be returned.'","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The core descriptive finding—that identical historical Search API queries yield highly variable video sets over time—is likely correct and is well controlled. My main concern is that the abstract and Section 4.2 over-interpret this finding as evidence of a specific popularity-based randomization mechanism, and the paper's own totalResults observation (Table 3) undercuts that interpretation. The authors should be given the opportunity to reframe the central claim; the descriptive results are worth publishing even with the mechanism claim substantially weakened. I also note that the manuscript cites the concurrent 'Forgetful by Design' audit (Rieder et al., arXiv:2506.11727) but does not compare its findings with that work; adding such a comparison would strengthen the paper's contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, the central descriptive result is real: identical historical queries issued at different dates return video sets that drift apart over time, and the error bars rule out deletions. That's a useful, well-documented finding for anyone collecting YouTube data via the Search API. Second, the mechanism the abstract advertises—popularity-based randomization that makes representative samples 'nearly impossible'—does not follow from the evidence, and the paper's own Table 3 points to a more mundane explanation.\n\nWhat's new: the 16-snapshot, 12-week, six-topic hourly-sliced design; the explicit non-ceiling test using per-hour maxima and the weak positive correlation between hourly volume and Jaccard similarity; and the comparison with ID-based endpoints (Video: list, CommentThreads) showing those are stable. The practical suggestions—split topics not time, watch totalResults, prefer ID endpoints—are concrete and likely to save researchers grief. The paper is transparent about its assumptions and flags the hourly-filtering uncertainty itself. That transparency earns it a fair read.\n\nSoft spots, in proportion. The hourly cohort assumption is load-bearing for the mechanism claim, and it looks wrong. Table 3 shows pageInfo.totalResults with modal 1,000,000 for three topics even though every query is scoped to a single hour. That means the API is building a result pool without applying the publishedAfter/Before bounds to the pool size; the time filters likely operate after the pool is formed, at daily or coarser granularity. If so, the zero-video hours and the non-peak scarcity are not evidence of deliberate withholding based on topical interest; they're just day-level sampling with a 500-video cap. The Jaccard decay in Figure 1 still stands—that's the key empirical contribution—but the 'empirical distribution' explanation in Section 4.2 and the 'nearly impossible' line in the abstract are not entailed. The paper notes the ambiguity in Section 3 and again in the totalResults discussion, so this is a matter of overinterpretation rather than fraud.\n\nOther notes. Six topics and one API client is a thin base for strong impossibility claims. The SHAP model has R²=0.19, so the feature-attribution story is suggestive, not mechanistic; the regressions in Appendix D are directionally consistent, which helps. The Higgs topic's stability is intriguing but could be confounded by age or pool size. No obvious citation problems; prior audits by Rieder et al. and McGrady et al. are cited and engaged with.\n\nWho this is for: anyone doing measurement on YouTube via the official API, and the IMC audience. It deserves a serious referee. I'd send it to review with a request to revise the abstract, reanalyze the hourly data in light of totalResults, and ideally validate on a small corpus where ground truth is available.\n\nRecommendation: engage.","headline":"Solid audit of YouTube Search API instability; the core variability result holds, but the popularity-randomization mechanism and 'impossible to sample' claims overreach the data, which the paper's own totalResults numbers undercut.","tokens_in":13803,"tokens_out":3177,"would_cite":true,"duration_ms":28745,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Running identical YouTube search queries over twelve weeks shows the API returns shifting historical video sets that track topical popularity around the query date.","keywords":["YouTube Data API","Search endpoint","API audit","result variability","historical sampling","replicability","data collection","empirical popularity distribution"],"falsifier":"Pick a historical hour with a small, known candidate pool, say ten videos, query it on many dates spanning high- and low-interest periods for the topic, and compare the returned sets: if the sets vary with query-date interest, the paper's mechanism is supported; if they are identical whenever the pool is below the API's per-page cap, the central claim is refuted.","tokens_in":12795,"feed_emoji":"📺","tokens_out":8425,"duration_ms":69479,"temperature":0.7,"pith_summary":"This paper tries to establish that YouTube's Search API is not a dependable tool for building historical video datasets: running the same query on different dates returns substantially different video sets for the same past period, and the variation tracks the topic's popularity around the query date rather than the API's 500-video ceiling. The evidence comes from a twelve-week audit that queried six topics hour-by-hour over 28-day historical windows and compared the returned video-ID sets. If the claim is right, researchers who split queries by time to bypass the 500-video cap are still getting a sampled, popularity-weighted view of history, not a complete one, and papers that report only their keywords and date ranges are missing a decisive variable. The paper also reports that the API tends to return shorter, more popular videos more often, while channel popularity matters less, and that smaller topic pools yield more consistent results. It closes with practical guidance: break topics into narrower sub-queries rather than splitting time, and prefer ID-based endpoints when completeness or replicability matters.","feed_headline":"YouTube Search API returns shifting videos for identical queries","feed_subtitle":"Twelve-week audit shows historical video sets decay with query date, undermining time-split data collection.","key_machinery":"The load-bearing machinery is the repeated identical-query audit design: for each of six topics, the same keyword query was sent for every hour of a 28-day historical window, and the whole 28-day sweep was repeated every five days for twelve weeks, producing 4,032 queries per sweep. The key quantities are Jaccard similarity, the size of the intersection of two video-ID sets divided by the size of their union, between successive sets $J(S_t, S_{t-1})$ and between the current and first sets $J(S_t, S_1)$, which quantify decay over time. A second-order Markov chain over the presence or absence of each video in successive collections reveals the rolling-window sampling dynamics, showing that a video is more likely to stay present or absent when its two previous states matched. The SHAP decomposition of a Poisson-objective gradient-boosting model identifies which video and channel features predict how often a video is returned, pointing to views, duration, and topic-pool size as the strongest predictors.","core_discovery":"The central claim is that the Search: list endpoint randomizes its results over time even for fully historical queries, making representative historical sampling effectively impossible, especially during non-peak topical periods. Concretely, Jaccard similarity between the video-ID set from the first collection and a later collection decays over the twelve weeks, and videos appear at time $t$ that were absent at $t-1$ even though the query window lies entirely in the past. The paper rules out ceiling effects: per-hour returned counts stay far below the 50-per-page maximum, and the correlation between hourly returned volume and set similarity is weakly positive, not negative. It interprets this as evidence that the API samples from an empirical distribution of topical interest, sometimes returning zero videos when the relative density of interest is low. A second-order Markov chain over presence and absence in successive collections shows videos drop in and out in a rolling-window fashion, and a SHAP analysis of a gradient-boosting model indicates that view count, shorter duration, and smaller topic pools are associated with more consistent returns.","pith_inferences":["If the API's sampling tracks query-period topical interest, then any dataset assembled from repeated search calls encodes the researcher's observation calendar as much as the historical event itself; longitudinal analyses should therefore model query date as a confound.","The stable ID-based endpoints provide a natural ground-truth check the paper does not run: randomly sample channels relevant to a topic, pull complete uploads via Channels:list and PlaylistItems:list, and compare that universe to search results to quantify how much of the historical record search misses.","The view-count and duration effects could be partly an artifact of the popularity-sampling mechanism rather than independent ranking criteria: shorter, popular videos may simply be overrepresented in the empirical interest distribution that the API appears to draw from.","A testable next step would be to run the same audit on a topic with a deliberately engineered upload schedule, uploading a known set of videos at known times, to recover the API's sampling weights directly."],"forward_implications":["A researcher who queries the same historical topic on two different dates may end up with two different video sets for the same day, so longitudinal datasets built from search snapshots inherit the query dates as a hidden variable.","Time-splitting a query into hourly bins does not defeat the API's sampling: per-hour returned counts stay far below the per-page maximum, and similarity does not drop where more videos are returned.","Smaller query pools produce more stable results, so breaking a topic into narrower sub-queries or adding AND terms is a more promising strategy than splitting the time window.","ID-based endpoints such as Video:list and CommentThreads:list show stable behavior, making channel-level discovery via Channels:list plus PlaylistItems:list a viable alternative when consistency matters.","The Markov-chain analysis implies videos enter and leave the returned set in rolling windows, so the set a researcher sees on a given day is a transient window over the API's internal distribution, not a cumulative archive."],"supporting_citations":[{"why":"Supplies the time-split query strategy that this paper tests and shows to be insufficient.","marker":"[24]"},{"why":"Documents the strong recency bias in time-split search queries, the direct baseline this audit builds on and contrasts with its own findings.","marker":"[25]"},{"why":"Establishes the 500-result cap and the per-page limits that motivate the ceiling-effect test and the hourly query design.","marker":"[28]"},{"why":"Shows YouTube's recommender uses an empirical distribution of video popularity against age, the pattern the paper cites as the analogous mechanism behind the API's sampling.","marker":"[4]"},{"why":"Documents that obtaining random samples from YouTube search is systematically difficult, framing the paper's motivation.","marker":"[16]"}],"fun_headline_variants":["YouTube API randomizes search results over time","Identical YouTube queries yield shifting video sets","Historical video samples skew with query date","YouTube Search API volatility blocks time-series research","Twelve-week audit finds YouTube API result drift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the API's publishedAfter and publishedBefore parameters genuinely limit the search to the requested one-hour window rather than returning videos sampled from a whole day and only then filtering them by the hour bounds.","fun_headline_variants_meta":{"raw":{"variants":["YouTube API randomizes search results over time","Identical YouTube queries yield shifting video sets","Historical video samples skew with query date","YouTube Search API volatility blocks time-series research","Twelve-week audit finds YouTube API result drift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1406,"prompt_tokens":903,"completion_tokens":503,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":519,"tokens_out":503,"duration_ms":5268,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:42:34.748796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick a historical hour with a small, known candidate pool, say ten videos, query it on many dates spanning high- and low-interest periods for the topic, and compare the returned sets: if the sets vary with query-date interest, the paper's mechanism is supported; if they are identical whenever the pool is below the API's per-page cap, the central claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the time-split query strategy that this paper tests and shows to be insufficient."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the strong recency bias in time-split search queries, the direct baseline this audit builds on and contrasts with its own findings."},{"cited_title":"part\": \"snippet","cited_arxiv_id":null,"evidence_quote":"Establishes the 500-result cap and the per-page limits that motivate the ceiling-effect test and the hourly query design."}],"review_version":1}