Towards an Automated Framework to Audit Youth Safety on TikTok

Francesco Corso; Francesco Pierri; Geng Liu; Linda Xue; Nicolo' Fontana; Stefano Ceri

REVIEW 4 major objections 5 minor 1 cited by

Towards an Automated Framework to Audit Youth Safety on TikTok

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An audit using scripted fake accounts on TikTok found that accounts declared as 13-year-olds and adults were shown nearly identical shares of videos that violate the platform's own youth-safety categories; active keyword searches raised the

desk verdict Useful preliminary audit of TikTok age-based moderation, but the headline null result is not yet firmly supported: the age-13 accounts are never verified as being treated as minors, and the main classifier has 59% precision. read the letter →

arxiv 2509.05838 v1 pith:L46YBOQI submitted 2025-09-06 cs.CY

Linda Xue , Francesco Corso , Nicolo' Fontana , Geng Liu , Stefano Ceri , Francesco Pierri This is my paper · ORCID

classification cs.CY

keywords TikTokaudityouthsafetycontentmoderationharmfulsockpuppetaccountsalgorithmicauditingLLMclassificationForYoufeed

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish whether TikTok's age-based safety measures actually change what minors are exposed to, using sockpuppet accounts registered as 13-year-olds and as adults in Italy. Over four days and more than 7,000 videos, passive For You scrolling produced nearly identical harmful-content estimates for both age groups, with median proportions around 3.4% for adults and 2.3% for youth. Active keyword searches raised harmful exposure to about 28% for both groups, suggesting that the platform's safeguards are weaker for search than for the recommendation feed. The paper also measures how well LLM and vision-language classifiers detect harmful content, reporting precision near 59%, and frames the pipeline as a repeatable way to audit a very large online platform. A reader should care because the result directly bears on whether age-based content moderation protects children in practice.

What carries the argument

The load-bearing mechanism is the paired sockpuppet audit: ten accounts created with a registered age of 13 and ten registered just above 18, run on TikTok's web interface in Italy, each with scripted sessions of passive scrolling and active keyword searching. The account pair is what lets the authors attribute any difference, or lack of difference, in the feed to age. The second mechanism is the label model: GPT-4o classifies each video as harmful or not based on the video description, using a category list taken from TikTok's own Community Guidelines, with Detoxify scoring comment toxicity and a 100-video manual annotation set used to check classifier precision.

What would settle it

Register fresh 13-year-old and adult accounts on TikTok mobile and web in Italy, and before collecting any feed data check whether the youth accounts exhibit known minor restrictions such as no direct messages, safety prompts, or limited search suggestions. If they do not, the near-identical feed exposure cannot be attributed to ineffective age moderation because the accounts may not be treated as minors at all.

Watch

Extended reading notes

Core claim

The paper's central claim is empirical: when fresh accounts are registered as 13-year-olds and as adults, and then scrolled through TikTok's For You Feed under identical automated sessions in Italy, the estimated share of videos that violate TikTok's own youth-safety categories is nearly the same. In passive scrolling, fewer than 10% of videos were classified as harmful for either group, with median proportions of 0.034 for adults and 0.023 for youth. Two youth accounts stood apart, with about 14% and 25% harmful videos, but the overall distributions overlapped heavily. When accounts actively searched harm-adjacent keywords, the harmful share jumped to roughly 28% for both adults and youth,

Load-bearing premise

The comparison hinges on TikTok actually applying youth-specific moderation to accounts that self-report age 13, but the paper does not verify this, for example by checking for restricted features or age verification on those accounts.

Editorial extensions

If this is right

If the central result holds, a self-reported age of 13 is not enough to change the mix of For You content on TikTok's web version in Italy; age-based filtering is not visibly differentiating the recommendation feed.
Active search for harm-related keywords is a stronger exposure pathway than passive scrolling, and it raises harmful exposure for adults and youth alike, so moderation policy and audits should focus on search.
Detecting harmful videos from the description alone yields precision around 59%, so automated audits need better multimodal classifiers before they can replace human annotation at scale.
Because the audit uses only public data and platform-visible accounts, the same pipeline can be rerun by external researchers or regulators to check compliance with the Digital Services Act's minor-protection requirements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: The near-null difference may be an artifact of self-reported age. If TikTok's web registration does not apply minor-specific restrictions to fresh accounts, the comparison tests account creation, not the platform's youth enforcement.
Editorial extension: The authors observed that English 'alcohol' was censored for youth accounts while Italian 'Alcol' still returned results. This suggests translation-blind moderation; an audit could map which harmful keywords are filtered across languages and find systematic gaps.
Editorial extension: Because the large-sample harm estimates rely on GPT-4o text-only labels, visuals that violate guidelines without textual clues are systematically undercounted; a multimodal relabeling could change the estimated prevalence and possibly the adult/youth difference.
Editorial extension: The two youth outlier accounts with 14% and 25% harmful shares suggest per-account variance may be driven by recommendation dynamics rather than age; a larger account sample could separate personalization noise from age effects.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take: this is a genuinely useful pilot study, not a decisive finding. It does something the literature needs—a systematic Italian TikTok audit with 20 fresh accounts, 7,000+ videos, passive and active protocols, and a manual annotation check on the classifiers. The result that adult and youth accounts see nearly identical harmful-content rates (medians about 3.4% vs 2.3%) is a new empirical datapoint that contradicts Eltaher et al.'s report for similar platforms. That contrast is worth knowing about.

What's good: the pipeline is clearly described and mostly reproducible; the authors are honest about their limitations; and they evaluate both text-only GPT-4o and video-based VideoLLaMA3 against human labels. The fact that VideoLLaMA3 doesn't beat text-only GPT-4o is a useful negative result for automated auditing.

The soft spots are load-bearing, not cosmetic. First, the adult-vs-youth comparison assumes TikTok treats an account registered as age 13 as a minor and applies youth-specific moderation. The paper never systematically verifies that—no checking of account settings, available features, or API fields. That said, the 'alcohol' vs 'Alcol' anecdote (English keyword censored for youth, Italian equivalent returning results) is some evidence that age-aware filtering is active, so the stress-test concern about a total setup artifact is partially mitigated. Still, a few minutes checking account state would have closed the loop.

Second, the classifier precision of 59% means the prevalence estimates are noisy; the near-identical distributions could reflect classifier bias rather than platform behavior. No confidence intervals or statistical tests accompany the central comparison, and the two youth accounts with 14% and 25% harmful rates are mentioned but never dissected—they might be outliers that materially change the story. The active-search component uses only one account per group, so the 28% vs 27.9% similarity is basically anecdotal.

Who this is for: researchers and regulators working on platform auditing and youth protection. It deserves a serious referee, but the verdict you'd expect is "revise"—validate the age treatment, add stats, and report per-account variation properly. If those hold up, it becomes a credible check on age-based moderation claims.

Recommendation: yes, engage. Send to peer review rather than desk reject, but referee it with real scrutiny on the account-state verification and the classifier evaluation.

Best.

Referee Report

4 major / 5 minor

Summary. This paper reports an observational audit of TikTok's age-based content moderation. The authors created 10 youth (age 13) and 10 adult (age >18) sockpuppet accounts in Italy, collected over 7,000 videos over four days via passive "For You Feed" scrolling and active searches with harm-adjacent keywords, and used GPT-4o on video descriptions to estimate harmful content prevalence, Detoxify to measure comment toxicity, and VideoLLaMA3 plus manual annotation to evaluate automated classifiers. The main finding is that harmful video prevalence during passive scrolling is low and similar across age groups (median 3.4% for adults vs 2.3% for youth; fewer than 10% overall), while active searching raises prevalence to roughly 28% in both groups. The authors conclude that TikTok's age-based moderation may not meaningfully differentiate exposure in practice.

Significance. If the central claim were supported, the paper would be a useful contribution to DSA-relevant platform accountability research: it uses a multi-day, multi-account sockpuppet design; covers both passive and active interaction modes; grounds harmful-content categories in TikTok's official community guidelines; and includes manual annotation for evaluating automated classifiers. The paper is also transparent about several limitations. However, the central comparison is not currently established because the age manipulation is unvalidated, the main classifier has low precision, the two youth-account outliers are ignored, and no statistical tests or confidence intervals are provided. These are fixable, but they are load-bearing for the headline conclusion.

major comments (4)

[Section 3.1 / Section 4.3] The central adult-vs-youth comparison rests on the assumption that setting an account's registered age to 13 causes TikTok to treat it as a minor. The paper never validates this manipulation: there is no check of Youth Safety Mode, age-restricted features, search filtering, or any platform-side marker. The observation that the Italian keyword "Alcol" still returned results for youth accounts is consistent both with absent age-based moderation and with the account not being recognized as a minor; the design cannot distinguish these. The null result in Figure 3 may therefore be a setup artifact rather than evidence about TikTok's moderation.
[Section 4.4 / Section 4.3] The harmful-prevalence estimates in Section 4.3 are produced by GPT-4o annotations of video descriptions, yet Section 4.4 reports that GPT-4o has precision of only 59% against manual labels on the 100-video validation set, and no recall is reported. When the estimated positive rate is only 2.3-3.4%, a classifier with 59% precision can substantially overstate true prevalence; the paper reports no correction, no confidence intervals, and no sensitivity analysis. The claim that fewer than 10% of videos are harmful is therefore not quantitatively supported by the evidence presented.
[Section 4.3 / Figure 3] Figure 3 shows two youth accounts with approximately 14% and 25% harmful proportions, several times the median for both groups. The text summarizes the result as "fewer than 10% of the videos were predicted to be harmful for both adult and youth accounts" and uses this to support the "minimal differences" conclusion, but it never analyzes or even discusses these outliers. Moreover, no statistical test is reported for the adult-vs-youth comparison. With N=10 accounts per group and high apparent variance, the conclusion requires at least an account-level test, effect size, or explicit treatment of outliers.
[Section 3.1 / Section 4.3] The active-search protocol uses only one Adult and one Youth account, but Section 4.3 reports aggregate prevalence rates of 28.44% (adult) and 27.91% (youth) and describes the increase over passive exposure as "significantly higher." No confidence intervals, per-account variability, or keyword-level breakdowns are given. With a single account per condition, the active-search findings cannot support RQ2 or the comparison between age groups.

minor comments (5)

[Appendix B, Table 2] The keyword table lists only 7 of the 13 harmful-content categories used in the classifier prompt (e.g., suicide/self-harm, sexual abuse, hate speech, and harassment are absent). Please clarify whether active searches actually covered all framework categories or only a subset, since this affects interpretation of the active-search results.
[Section 4.4] The manuscript says VideoLLaMA3 was tested on "a random selection of 100 videos" and then describes the manual sample as 50 per age group with 25 per group from keyword searches. Please clarify the exact sampling procedure and report the overlap between the classifier-evaluation set and the manual-evaluation set.
[Section 4.4] Precision alone is reported for GPT-4o and VideoLLaMA3. Please report recall, F1, and the confusion matrix against manual labels; without recall, the trade-off between missed harmful content and false alarms cannot be assessed.
[Section 3.1 / Results] Several claims would be easier to evaluate with explicit sample sizes: the number of videos collected per account/condition, the number of search videos per keyword, and the total N used for each figure. The current "over 7,000 videos" is too coarse.
[Text and references] Minor typographical and formatting issues include "FORYOUFEED(FYF)" (missing space), incomplete reference entries (e.g., WSJ Staff 2021), and the unresolved difference between the 13 categories in the prompt and the shorter keyword table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: observational audit; main limitation is unverified age-treatment manipulation, not circular reasoning.

full rationale

This paper is an observational audit of TikTok content exposure, not a mathematical derivation. The central adult-vs-youth comparison depends on the untested assumption that TikTok treats self-reported age-13 accounts as minors, but that is a construct-validity or manipulation-check limitation, not a circular definition. The harmful-content classifier is prompted with TikTok's own Community Guidelines, and the active-search keywords are also drawn from TikTok's Youth Safety guidelines; this may inflate the measured harm rate for keyword searches, but the paper does not present that rate as a derived prediction — it is an empirical measurement with acknowledged classifier limitations. GPT-4o's labels are evaluated against manual annotations (precision 59%), and the full-sample prevalence estimates are not calibrated to those labels, so they are not forced by the ground truth. The only self-citation (Corso et al. 2024) supports a usage statistic and is not load-bearing. No equation or fitted parameter reduces the conclusions to the inputs; the findings are therefore self-contained as an audit, with validity risks rather than circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper is an observational audit; it does not fit parameters or introduce entities. The central estimates depend on the three listed domain assumptions, especially the unverified age treatment and the weak classifier.

free parameters (1)

toxicity threshold = 0.5-0.7 (from Hua et al. 2020)
Used to classify a comment as toxic in Section 4.2; the choice range comes from prior work, not fitted to this dataset.

assumptions (3)

domain assumption TikTok applies youth safety moderation to accounts registered with age 13
The youth/adult comparison in RQ1 assumes that the platform recognizes age and enforces age-based content restrictions. Not verified in the paper.
domain assumption TikTok's web version For You Feed reflects mobile app recommendations
Data collected via web scraping; the authors note in Future Work that mobile interface should be used for ecological validity. This affects generalizability.
domain assumption GPT-4o text-only annotation of video descriptions reliably estimates harmful content
Used to produce prevalence estimates in Figure 3, but the validation on 100 videos shows precision of 59%, so the assumption is weakly supported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards an Automated Framework to Audit Youth Safety on TikTok." pith.science (2026). https://pith.science/paper/L46YBOQI

@misc{pith2026250905838,
  author       = {Pith},
  title        = {Pith review of: Towards an Automated Framework to Audit Youth Safety on TikTok},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L46YBOQI}},
  note         = {Machine review of arXiv:2509.05838}
}

read the original abstract

This paper investigates the effectiveness of TikTok's enforcement mechanisms for limiting the exposure of harmful content to youth accounts. We collect over 7000 videos, classify them as harmful vs not-harmful, and then simulate interactions using age-specific sockpuppet accounts through both passive and active engagement strategies. We also evaluate the performance of large language (LLMs) and vision-language models (VLMs) in detecting harmful content, identifying key challenges in precision and scalability. Preliminary results show minimal differences in content exposure between adult and youth accounts, raising concerns about the platform's age-based moderation. These findings suggest that the platform needs to strengthen youth safety measures and improve transparency in content moderation.

Figures

Figures reproduced from arXiv: 2509.05838 by the authors.

**Figure 2.** Distribution of toxicity for all comments (top) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Distribution of estimated proportion of harm [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Algorithmic Audit of Personalisation Drift in Polarising Topics on TikTok
cs.IR 2026-03 conditional novelty 6.0 of 10

TikTok personalises strongly, neutralises climate/vaccine/flat-earth content toward safe neutral topics, but sustains and often stance-reinforces US politics—with a tilt toward the oppose stance when both sides are seeded.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages · cited by 1 Pith paper

[1]

Fabian Baumann, Nipun Arora, Iyad Rahwan, and Agnieszka Czaplicka. 2025. Dynamics of algorithmic content amplification on tiktok. arXiv preprint arXiv:2503.20231

arXiv 2025
[2]

Maximilian Boeker and Aleksandra Urman. 2022. https://doi.org/10.1145/3485447.3512102 An empirical investigation of personalization factors on tiktok . In Proceedings of the ACM Web Conference 2022, pages 2298--2309

work page arXiv 2022
[3]

Francesco Corso, Francesco Pierri, and Gianmarco De Francisci Morales. 2024. What we can learn from tiktok through its research api. In Companion Publication of the 16th ACM Web Science Conference, pages 110--114

work page 2024
[4]

Fatmaelzahraa Eltaher, Rahul Krishna Gajula, Luis Miralles-Pechu \'a n, Patrick Crotty, Juan Mart \' nez-Otero, Christina Thorpe, and Susan McKeever. 2025. Protecting young users on social media: Evaluating the effectiveness of content moderation and legal safeguards on video sharing platforms. arXiv preprint arXiv:2505.11160

arXiv 2025
[5]

Jiaoju Ge, Yuepeng Sui, Xiaofeng Zhou, and Guoxin Li. 2021. Effect of short video ads on sales through social media: The role of advertisement content generators. International Journal of Advertising, 40(6):870--896

work page 2021
[6]

Yiqing Hua, Mor Naaman, and Thomas Ristenpart. 2020. https://doi.org/10.1145/3313831.3376548 Characterizing twitter users who engage in adversarial interactions against political candidates . In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, page 1–13, New York, NY, USA. Association for Computing Machinery

work page arXiv 2020
[7]

Mohsin Iqbal. 2022. https://uploads2.craft.co/uploads/operating_source/document/982548/57cb7f410c237c8e.pdf Tiktok revenue and usage statistics (2022) . Accessed: 2025-08-05

work page 2022
[8]

Richard Landis and Gary G

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159--174

work page 1977

Show all 13 references

[9]

Norvilitis, and Tian Xiao

Mingli Liu, Aixia Zhuang, Jill M. Norvilitis, and Tian Xiao. 2024. Usage patterns of short videos and social media among adolescents and psychological health: A latent profile analysis. Computers in Human Behavior, 151:108007

2024
[10]

Caroline Violot, Tugrulcan Elmas, Igor Bilogrevic, and Mathias Humbert. 2024. Shorts vs. regular videos on youtube: A comparative analysis of user engagement and content creation trends. In Proceedings of the 16th ACM Web Science Conference (WebSci ’24), pages 213--223, New Yo...

2024
[11]

WSJ Staff . 2021. https://www.wsj.com/articles/tiktok-algorithm-video-investigation-11626877477 Inside tiktok’s algorithm: A wsj video investigation . Accessed: 2025-08-05

2021
[12]

Cai Yang, Sepehr Mousavi, Abhisek Dash, Krishna P Gummadi, and Ingmar Weber. 2025. Studying behavioral addiction by combining surveys and digital traces: A case study of tiktok. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, pages 2106--2123

2025
[13]

Gummadi, Elissa M

Savvas Zannettou, Olivia-Nemes Nemeth, Oshrat Ayalon, Angelica Goetzen, Krishna P. Gummadi, Elissa M. Redmiles, and Franziska Roesner. 2024. https://arxiv.org/abs/2301.04945 Analyzing user engagement with tiktok's short format video recommendations using data donations . Prepr...

2024 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Fabian Baumann, Nipun Arora, Iyad Rahwan, and Agnieszka Czaplicka. 2025. Dynamics of algorithmic content amplification on tiktok. arXiv preprint arXiv:2503.20231

arXiv 2025

[2] [2]

Maximilian Boeker and Aleksandra Urman. 2022. https://doi.org/10.1145/3485447.3512102 An empirical investigation of personalization factors on tiktok . In Proceedings of the ACM Web Conference 2022, pages 2298--2309

work page arXiv 2022

[3] [3]

Francesco Corso, Francesco Pierri, and Gianmarco De Francisci Morales. 2024. What we can learn from tiktok through its research api. In Companion Publication of the 16th ACM Web Science Conference, pages 110--114

work page 2024

[4] [4]

Fatmaelzahraa Eltaher, Rahul Krishna Gajula, Luis Miralles-Pechu \'a n, Patrick Crotty, Juan Mart \' nez-Otero, Christina Thorpe, and Susan McKeever. 2025. Protecting young users on social media: Evaluating the effectiveness of content moderation and legal safeguards on video sharing platforms. arXiv preprint arXiv:2505.11160

arXiv 2025

[5] [5]

Jiaoju Ge, Yuepeng Sui, Xiaofeng Zhou, and Guoxin Li. 2021. Effect of short video ads on sales through social media: The role of advertisement content generators. International Journal of Advertising, 40(6):870--896

work page 2021

[6] [6]

Yiqing Hua, Mor Naaman, and Thomas Ristenpart. 2020. https://doi.org/10.1145/3313831.3376548 Characterizing twitter users who engage in adversarial interactions against political candidates . In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, page 1–13, New York, NY, USA. Association for Computing Machinery

work page arXiv 2020

[7] [7]

Mohsin Iqbal. 2022. https://uploads2.craft.co/uploads/operating_source/document/982548/57cb7f410c237c8e.pdf Tiktok revenue and usage statistics (2022) . Accessed: 2025-08-05

work page 2022

[8] [8]

Richard Landis and Gary G

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159--174

work page 1977