{"id":"f08197af-1197-4b31-b738-d285c7d3cb8d","arxiv_id":"2502.09573","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Zero-shot GPT-4o comes close to production classifiers on several video-moderation tasks, and prompt simplification plus decomposition-aggregation improves accuracy, though key gains are partly fitted to the test set.","lead":"This study tested how well GPT-4o can label TikTok videos into seven content-quality categories using only prompts, with no fine-tuning. It found that shortening long policy instructions and splitting a tricky category into sub-questions improves accuracy, which matters for large-scale video moderation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation-set leakage in the linear-regression aggregation (Exp. 3) is the load-bearing flaw: weights are fit on the same PR-evaluation data, so the claimed high-precision gain over baseline is an in-sample fit, not a zero-shot prediction.","rationale":"The reader's rejection is well-founded. The central claim is that prompt design alone substantially improves GPT-4o classification without fine-tuning. Two sub-claims support this: policy shortening (Exp. 2) and decomposition-aggregation (Exp. 3). The most load-bearing flaw is in Exp. 3: the 'best linear regression' weights are fit on the same data used to compute the precision-recall curves, so the reported high-precision advantage over the production model is an in-sample fit, not a zero-shot transfer result. The paper describes no data split for Exp. 3, and the phrase 'best linear regression' implies model selection that is not accounted for. Even if mean/max aggregation is parameter-free, the headline 'outperform the production model' is specifically attributed to the regression, making this a correctness risk rather than a mere presentation issue. The 96-word Sensitive result is also suspect: the prompt is not disclosed, Section 4.3 literally contains 'XXX words' while Figure 3 says '96 words', and no selection protocol is described. This is consistent with the reader's weaker assumption that the prompt may have been iteratively tuned on the evaluation data. A held-out evaluation with disclosed prompts and proper fitting would settle whether either claimed improvement transfers. The verdict remains REJECT; the paper could be resubmitted as a descriptive case study, not a validated finding.","tokens_in":8391,"tokens_out":2889,"duration_ms":26734,"concrete_test":"Split the clickbait dataset (>=500 videos) into training (e.g., 60%), validation (20%), test (20%). Use only the training split to fit the linear-regression weights over the eight GPT-4o subcategory scores; select any hyperparameters on validation; then compute the PR curve and AUC for the Reg aggregator on the held-out test split. Repeat with 5-fold cross-validation and report the mean and variance of the test PR curve. If the held-out Reg curve no longer dominates GPT-4o-single and the baseline at high precision, the Figure 4 claim is an artifact of fitting to the evaluation set. In addition, disclose the exact 96-word Sensitive prompt and the number of policy variants tried before the reported result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 Exp. 3 and Figure 4 report a 'best linear regression' of the eight subcategory scores (GPT-4o-multi Reg) that outperforms the single-prompt approach and beats the production model at high precision. The paper does not state that these regression weights were estimated on a separate training set; in fact, no train/test split for Exp. 3 is described anywhere, and the same balanced dataset is used for the PR curves. Because the regression coefficients are chosen to maximize discrimination on the same videos whose labels define the PR curve, the Reg curve is a training-set fit, not an out-of-sample prediction. The mean/max aggregators are parameter-free and still show gains, but the headline 'outperform the production model' claim rests specifically on the fitted regression. A second, related leak: the 96-word Sensitive policy is undisclosed and no search procedure is reported, so we cannot rule out that it was selected after seeing the Sensitive test results. However, the regression leakage alone is sufficient to invalidate the strongest claim as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates GPT-4o as a zero-shot classifier for seven TikTok feed-quality categories using balanced industry datasets. It reports AUC, false-positive and false-negative rates against production baselines, and proposes two prompt-engineering improvements: shortening a 4,023-word policy for Sensitive & Mature content to 96 words, and splitting the Clickbait policy into 8 subcategories whose scores are aggregated by mean, max, or linear regression. The paper claims that the shortened policy reduces false negatives and that the decomposition-aggregation approach, especially with linear regression, outperforms the single-prompt method and the production model at high precision.","tokens_in":8593,"tokens_out":4920,"duration_ms":46345,"significance":"The paper addresses a practical, high-impact problem (content moderation) and provides an evaluation framework based on real industry data. If the improvements are validated out-of-sample, the results would show that prompt design alone can yield meaningful gains without fine-tuning. The paper also has strengths: the experimental protocol is clearly described (temperature, sampling, balanced datasets), and the mean/max aggregators are parameter-free. However, the two headline claims are currently not supported by the reported experiments because both rely on choices that may be fitted to the evaluation data.","major_comments":[{"comment":"The precision-recall curve labeled 'Reg. GPT-4o-multi' is computed on the same balanced Clickbait dataset that was used to fit the linear regression weights. Since no train/test split or cross-validation is described, the curve is an in-sample fit, and the claim that this aggregator outperforms the production model at high precision is not an out-of-sample prediction. Please report cross-validated or held-out PR curves, or restrict the claim to the parameter-free mean and max aggregators.","section":"Section 4.3 Exp. 3 / Figure 4"},{"comment":"The shortened Sensitive & Mature policy is listed as 'XXX words' in the experimental setup and the actual 96-word prompt is not disclosed. The paper reports a reduction in FN of 0.07 and an AUC increase from 0.73 to 0.79, but without knowing the prompt or the procedure used to derive it, one cannot exclude the possibility that it was selected after inspecting the test-set results. Please disclose the full prompt and describe its construction as independent of the evaluation data.","section":"Section 4.3 Exp. 2 / Q3"},{"comment":"No error bars or significance tests are reported for any AUC or FP/FN comparisons in Table 1 and Figures 3–4. Given the dataset sizes of 'at least 500', differences such as 0.73 vs 0.79 AUC could be within sampling noise. Provide confidence intervals or a significance test for the key comparisons.","section":"Section 4.3 / Table 1"},{"comment":"The observed correlation between policy length and error type is based on only seven categories and is confounded by category difficulty. The paper itself acknowledges this in Q5, but the earlier phrasing in Q3 ('this pattern corresponds directly to the number of words in the policy') overstates the evidence. Please temper the causal language or provide a controlled analysis.","section":"Section 4.4 Q3–Q5"}],"minor_comments":[{"comment":"The placeholder 'XXX words' should be replaced with the actual word count; in Q3 the number 96 appears without explanation.","section":"Section 4.3 Exp. 2"},{"comment":"The terminology is inconsistent: the abstract says 'zero-shot' while Exp. 3 and Q6 use 'one-shot'. Please clarify the intended meaning and use it consistently.","section":"Section 3.2 / Abstract"},{"comment":"There is a typo: 'mutliple' should be 'multiple'.","section":"Section 1"},{"comment":"'SD/SF/Watermark' appears to be a typo; 'SD' should likely be 'SF' (Static Frame).","section":"Section 4.2 and Q2"},{"comment":"The formatting 'GP T− 4o' should be corrected to 'GPT-4o'.","section":"Figure 3 caption"},{"comment":"The phrase 'test the affect of shortening' should be 'test the effect of shortening'.","section":"Section 4.3 Exp. 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry case study with potentially interesting findings, but the current evaluation does not support the headline claims because of the in-sample regression fit and the undisclosed shortened prompt. The authors should be given the opportunity to provide held-out validation and full prompt disclosure. If the data cannot be re-analyzed or the prompts cannot be disclosed, rejection would be warranted. Also note that this is a workshop paper; the journal should assess whether the contribution meets its bar for novelty and rigor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful applied benchmark with one genuinely interesting empirical observation—long, detailed policies push GPT-4o into a conservative low-FP/high-FN regime—but the two headline prompt-engineering gains are not yet validated. Worth a serious referee, but only with mandatory held-out validation and disclosed prompts.\n\nWhat is actually new: a clean zero-shot comparison of GPT-4o against TikTok production classifiers on seven moderation categories, with policy word counts, AUCs, and FP/FN breakdowns. That table alone is useful for anyone deciding whether an LLM-only moderation baseline is viable. The policy-length finding on Sensitive & Mature is plausible and worth testing elsewhere. The decomposition-aggregation idea for clickbait is not new in general, but the specific eight-subcategory split and the comparison of mean, max, and regression aggregators is a reasonable empirical contribution.\n\nWhere it falls down: the 'best linear regression' in Exp. 3 is fit to the same balanced dataset used to draw the PR curves. No train/test split is described anywhere, so the Reg curve is an in-sample fit, and the claim that it beats the production model at high precision is not a zero-shot prediction. The mean and max aggregators are parameter-free and still beat the single prompt, so the decomposition idea survives, but the headline claim rests on the regression, and that regression is leaky. Second, the shortened Sensitive policy is never disclosed. We are told it is 96 words instead of 4023, but with no search procedure reported, we cannot rule out that it was chosen after seeing the test results. So the 0.73-to-0.79 AUC gain is an existence proof, not a validated, transferable recipe. Minor issues: no error bars, single category for the shortening effect, and the related work omits prior decomposition-prompt papers, so the word 'novel' is overstated.\n\nBottom line: the paper is honest in tone, the descriptive benchmark is solid, and the limitations section does not paper over the fact that prompt length alone does not explain everything. The load-bearing flaw is evaluation leakage in the regression plus the undisclosed prompt. I would not desk-reject it—there is real signal for practitioners—but I would send it back requiring a held-out evaluation with prompts disclosed and the regression fit on a separate training set.","headline":"Useful applied benchmark with one solid descriptive finding, but the two headline prompt-engineering gains are not yet supported because the regression weights are fit on the evaluation set and the shortened policy is undisclosed.","tokens_in":9114,"tokens_out":2114,"would_cite":false,"duration_ms":23208,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt design alone—compressing a 4,023-word moderation policy to 96 words and splitting clickbait into eight scored subcategories—raises zero-shot GPT-4o video classification to near-production level without fine-tuning.","keywords":["zero-shot video classification","GPT-4o","prompt engineering","policy compression","decomposition-aggregation","content moderation","feed quality classification","multimodal understanding"],"falsifier":"Run a preregistered replication in which the 96-word Sensitive policy is fixed before any evaluation and the clickbait regression weights are fit on a training split, then measure AUC and precision-recall on a held-out split of videos never used in prompt or weight selection; if the 0.73-to-0.79 AUC gain and the high-precision clickbait improvement disappear out of sample, the effects are fitting artifacts rather than transferable prompt-design gains.","tokens_in":8197,"feed_emoji":"🎬","tokens_out":8684,"duration_ms":64588,"temperature":0.7,"pith_summary":"This is an empirical study of whether GPT-4o, used as a zero-shot vision-language classifier, can handle real video content moderation across seven feed-quality categories defined by written policies. The paper's central claim is that prompt design alone, not fine-tuning, can substantially improve the model's accuracy. Two results carry the claim: compressing the 4,023-word Sensitive & Mature policy to 96 words reduced false negatives by 0.07 at only 0.01 more false positives, raising AUC from 0.73 to 0.79, and splitting clickbait detection into eight subcategories with separate scores, aggregated by mean, max, or linear regression, beat the single-prompt method, with the fitted aggregate outperforming the production model at high precision. The stakes for a reader are practical: if these effects are real, a platform can get meaningfully better moderation by rewriting instructions more tersely, at no training cost.","feed_headline":"Shorter policies lift GPT-4o's video AUC from 0.73 to 0.79","feed_subtitle":"Shorter policies plus split-then-merge scoring let zero-shot GPT-4o rival production classifiers, no retraining.","key_machinery":"The mechanism is the prompt itself: a system-level task description, the category policy, and an output contract that demands a JSON reply with reasoning and a 0-100 score, plus two prompt transformations applied to it. Policy compression rewrites a long, detailed moderation rule into a short statement of intent (4,023 words to 96 in the Sensitive case), which the analysis argues changes GPT-4o from conservative low-score behavior, with few false positives but many false negatives, to aggressive high-coverage behavior. Decomposition-aggregation splits one complex category into the subcategories named in its policy, collects an independent score for each, and combines them by mean, max, or fitted linear regression to form the final prediction. Both transformations operate on the instruction text only; the model weights, frames, and input features are untouched.","core_discovery":"On its own terms, the paper establishes three things. First, GPT-4o, given video frames and text features plus a written policy, matches production baselines on the simpler categories (Static Frame, Non-Interactive, Watermark) and trails them by no more than 0.1 AUC on Shocking, Useful, and Clickbait, with its worst performance on Sensitive & Mature (0.73 AUC, driven by a 0.41 false-negative rate). Second, policy length is a lever: shortening the Sensitive & Mature policy to 96 words raised AUC to 0.79, with false negatives dropping by 0.07 and false positives rising by only 0.01, a change the authors explain as a shift from conservative fine-brush scoring to aggressive broad-brush scoring. Third, for the 713-word Clickbait policy, decomposing the category into eight subcategories, asking GPT-4o to score each, and aggregating by mean, max, or linear regression outperforms single-prompt scoring, and the linear-regression aggregate beats the production baseline at high precision. The unifying claim is that application-specific prompt engineering is a cost-effective substitute for additional training.","pith_inferences":["An untested extension of the paper's logic is that long policies act as a prior over the score distribution, so policy compression should shift score entropy; one could test this by measuring GPT-4o's score spread as a function of policy length across several categories, which the paper does not report.","The threshold analysis suggests a calibration reading: the 96-word policy may not improve understanding of Sensitive content but rather shift the decision boundary, meaning a similar AUC gain could also be reached by re-ranking scores on a validation set.","The decomposition result points to a cheaper alternative to fine-tuning for other hard categories: if a complex policy's subcomponents can be enumerated, per-component scoring plus a learned aggregator may generalize better than a single prompt, but only if the aggregator is trained on separate data.","Because the authors attribute the effect to prompt properties rather than model-specific training, the same compression benefit should appear in other instruction-following vision-language models, a prediction the paper leaves untested."],"forward_implications":["If the results hold, a video platform can improve zero-shot classification accuracy by rewriting its moderation policies more tersely, without retraining models or collecting more labeled data.","Decomposition-aggregation gives a recipe for hard categories: break the policy into its natural subcategories, score each separately, and combine, with linear regression the strongest aggregator in the high-precision regime.","The 0.07 false-negative reduction at only 0.01 more false positives for Sensitive & Mature quantifies the practical trade between coverage and precision that prompt length controls.","GPT-4o's zero-shot parity with production models on several categories suggests it could serve as a fast, general fallback classifier for categories too small to train dedicated models.","The authors' observation that long policies push GPT-4o to be conservative yields a design rule: match prompt specificity to the error cost structure of the task."],"supporting_citations":[{"why":"Supplies GPT-4, the model whose zero-shot multimodal classification is being tested.","marker":"Achiam et al. (2023)"},{"why":"Defines the API prompting protocol the experiments follow for querying GPT-4o.","marker":"OpenAI (2025)"},{"why":"Frames the premise that large language models can be adapted to video understanding tasks.","marker":"Tang et al. (2023)"},{"why":"Provides the multi-modal classification framing (video, audio, text) the paper adopts.","marker":"Wang et al. (2017)"},{"why":"Represents the contrastive vision-language models that require fine-tuning, the alternative the paper positions against.","marker":"Radford et al. (2021)"},{"why":"Supplies the domain background of automated inappropriate-content detection that the tasks build on.","marker":"Yousaf & Nawaz (2022)"}],"fun_headline_variants":["Split-then-merge prompts beat single-shot on GPT-4o video","Zero-shot GPT-4o rivals production video classifiers","Prompt tweaks lift GPT-4o video AUC 0.73 to 0.79","Decompose, then aggregate prompts for GPT-4o video tagging","No retraining: prompt fixes boost GPT-4o video accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains stand or fall on the assumption that the shortened policy and the clickbait aggregation weights were not chosen by looking at the evaluation data: the 96-word policy is not reproduced in the paper, and the best linear regression weights are fit to the same dataset whose precision-recall curves are used to measure the gain.","fun_headline_variants_meta":{"raw":{"variants":["Split-then-merge prompts beat single-shot on GPT-4o video","Zero-shot GPT-4o rivals production video classifiers","Prompt tweaks lift GPT-4o video AUC 0.73 to 0.79","Decompose, then aggregate prompts for GPT-4o video tagging","No retraining: prompt fixes boost GPT-4o video accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000625,"raw_usage":{"total_tokens":2865,"prompt_tokens":890,"completion_tokens":1975,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1878}},"tokens_in":506,"tokens_out":1975,"duration_ms":152682,"temperature":1.0,"reasoning_tokens":1878,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:57:03.054976+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a preregistered replication in which the 96-word Sensitive policy is fixed before any evaluation and the clickbait regression weights are fit on a training split, then measure AUC and precision-recall on a held-out split of videos never used in prompt or weight selection; if the 0.73-to-0.79 AUC gain and the high-precision clickbait improvement disappear out of sample, the effects are fitting artifacts rather than transferable prompt-design gains.","supporting_citations":[{"cited_title":"Openai official api documentation, 2025","cited_arxiv_id":null,"evidence_quote":"Defines the API prompting protocol the experiments follow for querying GPT-4o."},{"cited_title":"Truly Multi-modal YouTube-8M Video Classification with Video, Audio, and Text","cited_arxiv_id":"1706.05461","evidence_quote":"Provides the multi-modal classification framing (video, audio, text) the paper adopts."},{"cited_title":"and Nawaz, T","cited_arxiv_id":null,"evidence_quote":"Supplies the domain background of automated inappropriate-content detection that the tasks build on."}],"review_version":1}