Auditing Algorithmic Personalization in TikTok Comment Sections

Referee: [Methods] Methods section: Validation of the 17 sock-puppet accounts is performed exclusively on For You Page video recommendations. No parallel validation, control condition, or divergence metric is reported for the comment-section rankings themselves. Consequently, it remains possible that observed between-group ranking differences arise from non-political behavioral traces (watch duration, like patterns, or engagement levels) introduced during training rather than from ideological signals.

Authors: We appreciate this observation. The sock-puppet training protocol was designed to embed ideological signals through repeated interactions (e.g., selective liking, commenting, and viewing of partisan content), and the FYP validation confirms that these signals successfully shifted the recommendation distribution toward left- or right-leaning content. Because comment-section ranking is governed by the same underlying personalization engine, we view the FYP results as a reasonable proxy. Nevertheless, we acknowledge that direct validation of comment rankings was not performed and that non-ideological behavioral traces could contribute to the observed divergence. In the revised manuscript we will add an explicit limitations paragraph discussing this possibility, including suggestions for future work that could include comment-specific validation metrics or control conditions. revision: yes

Referee: [Results] Results section: The statement that ranking divergence “is strongly correlated” with comment volume, engagement inequality, and partisan skew is presented without correlation coefficients, confidence intervals, or specification of the statistical procedure. Because these correlations are used to support the claim that personalization effects are context-dependent, the absence of quantified strength and robustness checks weakens the evidential basis.

Authors: We agree that the current presentation lacks the necessary quantitative detail. In the revised Results section we will report the exact correlation coefficients (Pearson or Spearman, as appropriate), 95% confidence intervals, p-values, and the precise statistical procedure used. We will also include any robustness checks (e.g., sensitivity to outlier videos or alternative divergence metrics) that were performed. These additions will allow readers to assess the strength and reliability of the reported relationships directly. revision: yes

Referee: [Methods] Methods and Results: The selection criteria for the 65 videos are described only at a high level (“politically neutral” yet containing “abundant discussions from both sides”). No explicit operationalization of neutrality, sampling frame, or checks for selection bias are provided, leaving open the possibility that the observed divergence patterns are artifacts of the particular video sample rather than general features of TikTok comment personalization.

Authors: We accept that the video-selection description requires greater precision. In the revised Methods section we will provide an explicit operational definition of “politically neutral” videos (videos from mainstream outlets without overt partisan framing, selected within a defined date range and meeting minimum view and comment thresholds), the sampling frame (e.g., stratified sampling across election-related topics), and the criteria used to confirm “abundant discussions from both sides” (e.g., keyword-based comment sentiment balance or manual coding of a subsample). We will also report any post-hoc checks for selection bias, such as comparison of engagement distributions against a larger pool of election videos. revision: yes