{"id":"6be9bc8b-8346-4340-bc6d-389608a511d9","arxiv_id":"2608.09917","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Three TikTok data collection tools retrieve largely non-overlapping datasets for identical queries, especially for hashtag and keyword searches.","lead":"This study compared three tools for collecting TikTok data and found that they return very different videos and comments for the same searches. Researchers should treat TikTok datasets as tool-dependent rather than neutral samples of the platform.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero Jaccard overlap may be a sampling artifact: with large query populations and quota-capped samples, independent random draws yield J≈0, so the headline divergence is not established by Table 2.","rationale":"The paper's declared central claim is that the three tools 'return vastly different results for the same queries.' The strongest numerical support is Table 2, where J=0.00 for hashtag and keyword endpoints. I read the paper in good faith: the collection is longitudinal, success rates are reported explicitly, and the API(R) pagination diagnostic is a useful replication attempt. But the inference from zero Jaccard to tool-dependent sampling has a statistical hole that the reader's verdict did not call out. In a large population with quota-limited independent samples, zero overlap is the default outcome, not evidence of systematic divergence. The paper's own Figure 6 notes some targets have billions of posts, so under the null the expected overlap for API(D) and API(R) is essentially zero. The same issue applies to API(D) versus Apify, especially because Table 5 shows their temporal windows are often disjoint; footnote 3's claim that both datasets contain overlapping time periods is not reconciled with Table 5's reported spans. This concern does not disprove the authors' conclusion—their engagement and temporal-span differences are suggestive, and the user/comments endpoints provide more meaningful overlap comparisons—but it determines whether the headline quantitative claim is as strong as stated. The proposed self-overlap and null-model test would settle whether J=0.00 is diagnostic of tool divergence or merely a consequence of sampling without replacement from a huge population. I therefore keep the reader's conditional verdict, while adding this specific technical condition to the path toward acceptance.","tokens_in":18296,"tokens_out":12692,"duration_ms":123322,"concrete_test":"Run a self-overlap and null-model check. (1) Pin the exact wrapper version and run API(R) twice on the same hashtag and keyword, same month/date window and same quota settings, then compute Jaccard between the two runs. If J≈0 within the same tool, then cross-tool J=0 is a sampling artifact. (2) For each of the eight query targets, estimate population size N from TikTok tag-page metadata (as in Fig. 6), simulate 1,000 pairs of independent quota samples with sizes matching Table 1/4 (e.g., n1=50k, n2=3k), and compare the observed J=0.00 to the simulated null distribution. If observed J falls within the null, the zero-overlap evidence for systematic divergence fails; if it is far below the null, the central claim is strengthened. Also report query-level J values with both tools restricted to a common temporal window to remove the age/sampling-frame confound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2's headline evidence for cross-tool divergence is pairwise Jaccard similarity. For hashtag and keyword endpoints, all reported J values are 0.00. But this is exactly what one expects if each tool draws a quota-capped sample without replacement from a very large matching population: expected overlap is n1·n2/N, and for queries like #fyp (N in the billions, per Fig. 6) the expected number of shared videos is below one even if the tools are behaviorally identical. The paper provides no null baseline, so J=0.00 cannot be interpreted as 'vastly different results.' The API(D)–API(R) comparison is especially problematic: API(R) is supposed to be random, and two random samples of roughly 50k and 4k from a huge population will almost never share a video. Thus the central quantitative claim—that different tools retrieve systematically different subsets—is not supported by the overlap numbers. The engagement and temporal-span differences are more informative, but they are confounded with video age and retrieval-window design (Table 5 shows API(D) windows as short as 6 minutes for high-volume queries, while Apify spans years), so they cannot independently carry the causal conclusion that tools are 'skewed by TikTok's algorithm.' A null model or self-overlap baseline is required before interpreting J.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a 30-day comparison of three TikTok data collection tools — the official TikTok Research API in two modes (API(D) and API(R)), Pyktok, and Apify — across five endpoints: User, Hashtag, Keyword, Comment, and Related Video. The authors measure collection success rates, temporal coverage, engagement distributions, and pairwise Jaccard overlaps, and they find large differences in volume (API(D) collected 442K hashtag videos versus Apify's 10K), temporal span (API(D) often returned only minutes-to-months of content while Apify spanned years), engagement profiles (Apify returned substantially higher-engagement videos), and near-zero overlap for most cross-tool comparisons. They conclude that tool choice strongly shapes TikTok datasets and offer transparency, methodological, and ethical recommendations.","tokens_in":18567,"tokens_out":6434,"duration_ms":57413,"significance":"If the findings hold, this is a valuable empirical contribution to the reproducibility debate in computational social science. The study is longitudinal, covers multiple endpoints, and uses publicly available tools; the Pyktok–Apify user-endpoint agreement (J=0.71) is a concrete and non-obvious positive result, while the API(D)–Apify engagement divergence (large rank-biserial effects in Figure 5) is a strong, falsifiable observation. At the same time, the overlap evidence is not statistically calibrated, and some claims in the abstract and conclusion exceed what the data can support. The paper would be a useful reference for TikTok researchers, but it needs revision before the central claim can be accepted as stated.","major_comments":[{"comment":"The interpretation of J=0.00 for the Hashtag and Keyword endpoints as evidence of systematic divergence lacks a null baseline. With quota-capped samples drawn from very large matching populations (e.g., #fyp has billions of posts, per Figure 6), two tools sampling uniformly from the same population would also be expected to produce near-zero Jaccard overlap because the expected overlap is approximately n1*n2/N. The paper does not report expected overlap under independence, a self-overlap baseline (same tool run twice), or a repeated-sampling diagnostic. The J=0.00 values therefore cannot by themselves establish the headline claim of 'vastly different results.' The temporal-coverage and engagement analyses (Table 5 and Figures 4–5) are more informative and should carry the argument, or the authors should add a null model.","section":"Data Coverage and Overlap by Tools, Table 2"},{"comment":"The abstract states that 'the three tools yielded comprehensive and consistent results only for the user endpoint,' but Table 2 reports API(D)-Pyktok J=0.06 and API(D)-Apify J=0.03 for the User endpoint, with Pyktok-Apify J=0.71. The user endpoint is therefore not consistent across all three tools; rather, the two scraping tools agree with each other and both diverge from the Research API. This wording should be corrected, as the current phrasing overstates the degree of agreement.","section":"Abstract and Conclusion"},{"comment":"The diagnostic establishing that API(R) suppresses pagination is based on a single test on April 24, 2026, and the manuscript does not report exact versions or commit hashes of the TikTok API wrapper, Pyktok, or the Apify actors. Since the paper's central theme is reproducibility, and since the API(R) behavior is used to explain the API(D)-API(R) overlap results, the authors should either pin the exact software versions and provide repeated diagnostics, or explicitly limit the pagination claim to the observed period and configuration.","section":"Data Collection and Appendix Table 6"},{"comment":"The conclusion states that 'when using front-end scraping techniques (e.g., Apify and Pyktok), data is skewed by TikTok's algorithm,' but the Discussion correctly cautions that without ground truth the observed pattern should be interpreted as 'divergence across retrieval strategies rather than as direct evidence that any one tool is biased relative to the full TikTok population.' The conclusion should be reworded to match this caveat, since the engagement differences could reflect API underrepresentation, Apify overrepresentation, or both.","section":"Conclusion"}],"minor_comments":[{"comment":"The phrase 'the hyperparameter is random' is garbled; it should refer to the is_random parameter throughout.","section":"Data Collection"},{"comment":"The paper states that no new code, data, or software will be released; for a methods-comparison paper, releasing the exact query scripts and configuration files would substantially strengthen the reproducibility recommendations.","section":"Ethics Checklist"},{"comment":"Table 1 omits API(R) rows; the text should explicitly note that API(R) was introduced only in the second collection cycle and is therefore excluded from the success-rate table.","section":"Table 1"},{"comment":"The reference 'Meza, R.; Mogos, , A.-A.; and Prundaru, G. 2023' contains a stray comma in the author list, and 'sockpuppetaccounts' appears without a space in the Related Work section; these should be corrected.","section":"References and Related Work"}],"recommendation":"major_revision","confidential_remarks":"One author (Deen Freelon) is the developer of Pyktok, and the paper recommends Pyktok for the user and comment endpoints. The manuscript does not include an explicit conflict-of-interest/self-evaluation disclosure. I would ask the editor to require such a disclosure before publication. The paper fits the conference's scope as a methods-audit contribution, but the absence of a null model for the overlap statistics and the abstract's overclaim are the main reasons for the major-revision recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version: this paper is worth taking seriously but needs to be reined in. The cleanest contribution is the pagination bug in the official TikTok Research API wrapper when is_random=True. They show that the wrapper stops after the first page because continuation fields come back empty, and they reproduce it in an April 2026 diagnostic. That is a concrete, actionable finding.\n\nThe systematic comparison itself is a good idea. Five endpoints, three tools, matched queries over 30 days, with an honest limitations section. I credit them for not pretending there is a ground truth and for framing the results as divergence, not bias.\n\nWhere I part ways is the strength of the central claim. The abstract says the three tools return 'vastly different results,' but the headline evidence, Table 2, is pairwise Jaccard. For hashtag and keyword endpoints, all J values are 0.00. As you know, that is exactly what you expect when two quota-capped tools independently sample without replacement from a massive population — #fyp has billions of posts. Expected overlap is n1*n2/N, which is essentially zero even if the tools were behaviorally identical. Without a null model or self-overlap baseline (e.g., same tool run twice), J=0.00 cannot be read as systematic divergence. The stress-test note is right about this.\n\nThe user endpoint numbers also contradict the abstract. J=0.06 and 0.03 between the API and the scrapers are not 'comprehensive and consistent results.' Low overlap is still the story, just less dramatic. The paper also claims 'all conditions held constant' when they actually varied is_random, quotas, and retrieval windows across tools.\n\nThe engagement and temporal-span differences are more informative, but they are confounded with tool architecture. API(D) pulls recent low-engagement videos; Apify scrapes front-end pages and gets older, higher-engagement content. You cannot infer that TikTok's algorithm is 'skewing' anything without a controlled comparison. Still, these differences are real and worth reporting as descriptive facts.\n\nMissing code/data is a real problem for a methods paper. The ethics checklist says they don't release new assets, but the whole point is that the field needs replication. They should at least release query logs and the pagination diagnostic script. Also, one author is the Pyktok developer. They mention it in passing, but they don't discuss how that shapes the evaluation. That needs an explicit statement.\n\nBottom line: a serious referee should see this, but it needs major revision. I'd ask for a null model, a toned-down abstract, artifact release, and a COI disclosure. If you read it as a descriptive audit with a documented bug and careful recommendations, it is useful. As a demonstration of tool-induced bias, it doesn't yet hold up.","headline":"Valuable audit with a real pagination find, but the headline divergence claim overreaches the overlap numbers.","tokens_in":70,"tokens_out":2252,"would_cite":true,"duration_ms":30188,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Same TikTok queries give vastly different data across three tools","keywords":["TikTok data collection","Research API","web scraping","reproducibility","computational social science","algorithmic curation","Jaccard similarity","data acquisition tools"],"falsifier":"Re-running the same matched collection protocol on a different set of hashtags and keywords (for example, health and entertainment terms not used here) during the same month and finding high Jaccard overlap, say above 0.5, between the Research API and Apify results would contradict the paper's claim of systematic divergence; likewise, showing that the API wrapper's is_random=True mode paginates correctly in a second diagnostic with a different wrapper version would weaken the pagination finding.","tokens_in":18115,"feed_emoji":"📊","tokens_out":4981,"duration_ms":40762,"temperature":0.7,"pith_summary":"This paper tries to establish that the choice of a TikTok data collection tool is not neutral: the official Research API, Pyktok, and Apify return largely non-overlapping datasets for identical hashtag, keyword, comment, and related-video queries. Only the user endpoint produced comparable results across all three tools. Because the official API pulls from back-end endpoints while Pyktok and Apify scrape what TikTok's front end displays, the tools retrieve different time periods and popularity levels, so conclusions about TikTok content can change depending on which tool built the dataset. The authors argue this undermines reproducibility and generalizability in TikTok research, and they use the divergence to argue for detailed transparency in data collection reporting.","feed_headline":"Same TikTok queries give vastly different data across tools","feed_subtitle":"A 30-day audit finds the official API and scraper tools rarely return the same videos, threatening reproducibility.","key_machinery":"The comparison rests on a standardized longitudinal collection pipeline: the same users, hashtags, keywords, and seed videos were queried daily for 30 days (August 1-30, 2025) in repeated 10-day cycles, with tools queried on matched schedules. The key analytic objects are pairwise Jaccard similarity coefficients for overlap, success rates defined as collected-to-goal ratios, temporal span of returned videos, and engagement distributions compared with Wilcoxon rank-sum tests. The underlying mechanism that drives the divergence is the access pathway: the Research API calls back-end endpoints, while Apify and Pyktok scrape TikTok's front-end pages, so the latter inherit the platform's algorithmic curation and visibility filters. The paper also treats the is_random parameter of the Research API wrapper as an object of investigation, showing through a diagnostic that it changes pagination behavior and thereby sample size.","core_discovery":"The central discovery is a systematic divergence among the three tools under matched query conditions. For hashtag and keyword searches, pairwise Jaccard similarity between any two tools was 0.00, meaning no videos were shared; the Research API's two modes (is_random False and True) also returned disjoint video sets while exhibiting similar engagement profiles. The Research API tended to return large volumes of low-engagement videos from narrow, recent time windows, whereas Apify returned small sets of high-engagement videos spanning years, because it scrapes the algorithmically curated pages that real users see. For comments, Pyktok retrieved the most comments per video, and pairwise comment overlap ranged from 0 to about 40 percent. A diagnostic test further showed that the API wrapper's is_random=True mode fails to paginate: responses carry has_more=False and no cursor, so collections stop after roughly 10 percent of the documented target. The authors interpret these results as evidence that no current tool can produce truly random samples of TikTok, and that tool-specific sampling confounds threaten the validity of single-tool studies.","pith_inferences":["If the divergence is stable, meta-analyses of TikTok studies should model the collection tool as a moderator or covariate; otherwise cross-study effect sizes confound platform effects with tool effects.","A testable extension would benchmark each tool against a ground-truth corpus of videos verified directly from creators (e.g., via official metadata or creator-provided archives) to estimate which tool, if any, comes closest to the full platform population.","The API(R) pagination finding suggests that researchers who used is_random=True should audit their logs for has_more and cursor values; datasets collected without pagination are identifiable and could be re-weighted or re-collected.","Because TikTok changes its front end frequently, the observed zero-overlap pattern may drift over time; repeating this audit at later dates would show whether the divergence is a stable property or a snapshot."],"forward_implications":["Researchers who generalize from a single TikTok data collection tool risk describing that tool's retrieval behavior rather than the platform's content.","Previous TikTok studies based on one tool may need re-examination, since a different tool could have produced different conclusions for the same queries.","Front-end scraping tools (Apify, Pyktok) skew toward algorithmically recommended, high-engagement content; the Research API skews toward lower-engagement videos and query-dependent recent windows.","Complete samples were achieved only for the user endpoint; hashtag, keyword, comment, and related-video collections are partial and tool-dependent.","The undocumented pagination failure of the API's is_random=True mode means studies using that setting may have collected first-page-only results, about 10 percent of intended targets."],"supporting_citations":[{"why":"Documents regional and temporal biases in TikTok Research API data, providing prior evidence that API samples deviate from the platform population.","marker":"Corso, Pierri, and De Francisci Morales 2024"},{"why":"Shows some publicly available TikTok videos are missing from API results, establishing that API output is incomplete relative to the web.","marker":"Pearson et al. 2025"},{"why":"Demonstrates that TikTok's stated algorithmic explanations often do not match user behavior, supporting the paper's claim that sampling processes are opaque.","marker":"Mousavi, Gummadi, and Zannettou 2024"},{"why":"Frames the post-API research landscape in which multiple data access tools coexist, motivating the need for cross-tool comparison.","marker":"Freelon et al. 2025"},{"why":"Reports structural limitations of the Research API including rate limits and inconsistent quotas, informing the paper's analysis of data loss.","marker":"Luceri et al. 2026"},{"why":"Identifies personalization factors that shape TikTok feeds, explaining why front-end scraping results are algorithmically curated.","marker":"Boeker and Urman 2022"}],"fun_headline_variants":["TikTok data tools rarely agree, audit finds","Zero overlap: TikTok hashtag results vary by tool","Official API vs scrapers: TikTok data mismatch","Three TikTok tools return disjoint datasets","TikTok research tools fail reproducibility test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The observed cross-tool differences reflect stable properties of the tools under typical use, rather than transient TikTok updates or errors in this particular configuration, and the four users and four query terms tested stand in for TikTok research generally.","fun_headline_variants_meta":{"raw":{"variants":["TikTok data tools rarely agree, audit finds","Zero overlap: TikTok hashtag results vary by tool","Official API vs scrapers: TikTok data mismatch","Three TikTok tools return disjoint datasets","TikTok research tools fail reproducibility test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1452,"prompt_tokens":957,"completion_tokens":495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":573,"tokens_out":495,"duration_ms":4654,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:24:34.943578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same matched collection protocol on a different set of hashtags and keywords (for example, health and entertainment terms not used here) during the same month and finding high Jaccard overlap, say above 0.5, between the Research API and Apify results would contradict the paper's claim of systematic divergence; likewise, showing that the API wrapper's is_random=True mode paginates correctly in a second diagnostic with a different wrapper version would weaken the pagination finding.","supporting_citations":[],"review_version":1}