{"id":"c8065f6e-ff1c-4a1f-b7ff-2aaccae33e9a","arxiv_id":"2509.04714","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Claude 3.5 Sonnet, prompted with thumbnails, subtitles, and video summaries, detects misleading YouTube thumbnails with up to 93.8% accuracy on a new cross-country dataset.","lead":"This paper tests whether large language models can decide if a YouTube thumbnail is misleading by comparing it with subtitles and an automatic summary of the video. On a new 2,843-video dataset spanning eight countries, Claude 3.5 Sonnet reached 93.8% accuracy, suggesting LLMs could help automate thumbnail moderation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1 comparison with CHECKER is invalid: Claude's 0.7227 and CHECKER's 0.7153 come from different datasets (2,843 vs 197 videos); 'match or exceed' is unsupported unless both systems run on a shared test set.","rationale":"The paper has two pillars: (1) a new cross-cultural dataset and an LLM pipeline that achieves high accuracy on it; (2) a claim that this prompt-based approach matches or beats a specialized supervised detector, CHECKER, without training. The second pillar is essential to the paper's stated contribution and to its framing in the abstract and introduction. The comparison as reported is between F1 scores computed on different test sets. CHECKER's 0.7153 is cited from the original paper's 197-video test set; Claude's 0.7227 is computed on the authors' 2,843-video dataset. No shared evaluation is described. Because F1 depends on class distribution and task difficulty, the two numbers cannot be compared. This is not a question of annotation subjectivity or confidence intervals; it is a direct logical gap in the central claim. If the comparison is removed or relabeled, the remaining contribution is a dataset and a set of absolute performance numbers, not a state-of-the-art comparison. A same-dataset evaluation is a feasible and concrete fix, so the appropriate verdict is CONDITIONAL rather than REJECT. The reader's weakest assumption focused on gold-standard validity; this is a separate, more directly falsifiable flaw, though the reader did note that the CHECKER protocol was undocumented.","tokens_in":19359,"tokens_out":5731,"duration_ms":54538,"concrete_test":"Obtain or reimplement CHECKER and evaluate it on the ThumbnailTruth dataset using the same test protocol and inputs (noting that CHECKER uses thumbnail + title, so supply titles where needed); also evaluate Claude 3.5 Sonnet + dynamic few-shot on CHECKER's 197-video test set. If Claude's F1 remains above CHECKER's on a shared test set, the comparative claim is supported; otherwise Table 3 must be relabeled as cross-dataset and the 'match or exceed' claim removed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central comparative claim in Section 5 and Table 3 is not supported by the reported numbers. Claude 3.5 Sonnet + Dynamic Examples has F1 = 0.7227, computed on the ThumbnailTruth dataset of 2,843 videos (1,359 MTV, 1,484 NMTV). CHECKER + Block (τ=0.30) has F1 = 0.7153, taken from CHECKER's original 197-video test set (64 clickbait, 133 non-clickbait), as stated in the paper. These are different evaluation sets with different class priors (~48% vs ~32.5% positive), different content distributions, and different annotation protocols. A model can score 0.72 on one benchmark and 0.71 on another for reasons unrelated to quality. The paper never reports running CHECKER on ThumbnailTruth, nor running Claude on CHECKER's test set. Therefore the headline 'Claude matched or outperformed CHECKER on every metric' and the conclusion that 'prompt-based LLMs can reach state-of-the-art accuracy without task-specific model training' do not follow from the data. F1 values are not portable across benchmarks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ThumbnailTruth, a dataset of 2,843 YouTube videos from eight countries, with 1,359 videos labeled as having misleading thumbnails and 1,484 as non-misleading, along with a codebook, scripts, and a subset of data made public. The detection pipeline combines thumbnail images, subtitle transcripts, and generated video-to-text descriptions, and is evaluated with four multimodal LLMs (Claude 3.5 Sonnet, Gemini 1.5 Flash, GPT-4o, GPT-4o Mini) under zero-shot, fixed few-shot, and dynamically retrieved few-shot prompting. The headline result is that Claude 3.5 Sonnet with dynamic few-shot prompting reaches 93.8% accuracy, with high precision and recall. The paper also compares the best Claude configuration to CHECKER, a supervised multimodal detector, reporting an F1 of 0.7227 versus 0.7153, and concludes that prompt-based LLMs can match or exceed state-of-the-art task-specific models without training. Additional analyses cover per-category performance, cultural/geographic variation, cost, and a modality ablation.","tokens_in":19629,"tokens_out":5471,"duration_ms":54308,"significance":"If the dataset and evaluation protocol are accepted, the paper makes a useful contribution: a publicly released cross-cultural dataset for a real content-moderation problem, a reproducible multimodal LLM pipeline, and systematic prompt ablations. I credit the authors for releasing the dataset, codebook, and scripts, and for evaluating multiple commercial LLMs under a consistent input setup. However, the central comparative claim against CHECKER is not supported by the data as reported, the ablation table contradicts the accompanying text, and the evaluation protocol has a transductive flavor that makes the headline accuracy hard to interpret as a prospective detection rate. These issues are load-bearing for the paper's main claims, so the manuscript needs substantial revision rather than minor polishing.","major_comments":[{"comment":"The F1 comparison with CHECKER is invalid as presented. Claude 3.5 Sonnet + Dynamic Examples has F1 = 0.7227 computed on the ThumbnailTruth dataset (2,843 videos, ~48% positive), while CHECKER + Block (τ=0.30) has F1 = 0.7153 taken from the original CHECKER 197-video test set (64 clickbait, 133 non-clickbait, ~32.5% positive). These are different evaluation sets with different class priors, content distributions, and annotation protocols. F1 values are not portable across benchmarks, so the statement that 'Claude matched or outperformed CHECKER on every metric' and the conclusion that prompt-based LLMs can reach state-of-the-art accuracy without task-specific training do not follow. Both systems must be run on a shared held-out test set, or the comparison must be removed/reframed as indicative only.","section":"Section 5, Table 3"},{"comment":"The ablation table contradicts the text. The paper states that 'the best overall performance is observed in the full-input setting, Claude-Zero Shot', but the table shows full-input accuracy 0.8920, which is below ABL-NS (0.9076) and ABL-ND (0.9077); the full-input recall 0.8430 is also below ABL-ND (0.8987) and ABL-NS (0.8856). The likely explanation is that incomplete responses (refusals) were excluded differently across rows, so metrics are not computed on a comparable subset. The authors should report the number of excluded cases per configuration, recompute metrics on the common subset, and correct the claims about complementarity and the superiority of the full input.","section":"Section 6, Table 4"},{"comment":"The evaluation protocol is transductive and the headline accuracy lacks uncertainty quantification. Dynamic few-shot examples are retrieved from the same dataset by semantic similarity to the input video's description, and the best prompt is selected after seeing the full evaluation dataset. This means the reported 93.8% is not a prospective or train/test estimate and cannot be directly compared with supervised models evaluated on separate test sets. The paper should use a held-out split for prompt selection and retrieval, state explicitly whether ground-truth labels of test instances are used anywhere in prompt construction, and report confidence intervals or significance tests for the differences in Table 1 (e.g., 93.8% vs 84.8%).","section":"Section 2.4 and Section 4.2"},{"comment":"The gold standard is the subjective judgment of two annotators (one author and one graduate student), with Cohen's kappa 0.9633. All reported accuracy, precision, recall, and specificity numbers measure agreement with these two annotators' codebook, not with an independent platform or audience ground truth. Given the paper's cross-cultural claims, this is a load-bearing concern: the 93.8% figure may reflect inter-rater agreement with the codebook rather than detection of 'actually misleading' thumbnails. The authors should provide an external validation (e.g., additional annotators from each country, or comparison with platform outcomes), or at minimum report per-country disagreement rates and borderline-case resolution.","section":"Section 2.2"}],"minor_comments":[{"comment":"The text says country classification used the UN World Economic Situation and Prospects 2024 report, but reference [5] is a Statista URL about YouTube users. The citation does not match the source described.","section":"Section 2.1"},{"comment":"Typo: 'We acknowledge that certain video types. particularly...' should be 'certain video types, particularly...'.","section":"Section 3.2"},{"comment":"The per-category table reports accuracy and F1 on a 'balanced subset' but does not give the number of videos per category. Without support counts, the F1 values are hard to interpret, especially for categories with small N.","section":"Table 2"},{"comment":"The caption says 'Average Number of Videos Processed by Each Model' but the rows appear to be total counts (e.g., Claude processed 2,759 videos). Clarify whether these are counts or averages, and report the denominator for the full dataset.","section":"Table 6"},{"comment":"The subsection titled 'Limitations and Areas for Improvement' discusses only model weaknesses (celebrity recognition, etc.). The paper should also include a limitations subsection covering dataset biases, annotation subjectivity, and the transductive evaluation issues raised above.","section":"Section 6"},{"comment":"The text refers to an 'anonymized GitHub repository', but Section 1 gives a non-anonymized repository URL and the author names are present. For a double-blind review, this should be anonymized consistently, or the wording should be changed.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The dataset release is a genuine contribution and the paper has a useful core idea, but the CHECKER comparison is not supportable as currently reported and the ablation text/table mismatch is conspicuous. If the authors can rerun both systems on a shared evaluation set, the comparison claim becomes credible; otherwise, removing the direct comparison would be more honest. I would not reject outright because the dataset and pipeline could be valuable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it builds something real and then overclaims on top of it. The new dataset—2,843 YouTube videos from eight countries, balanced between misleading and non-misleading, with a written codebook and near-perfect inter-annotator agreement (kappa 0.96)—is a genuine contribution. If you do any work on clickbait or content moderation, that dataset alone is worth having. The central empirical finding is also plausible: Claude 3.5 Sonnet with dynamic few-shot prompting gets 93.8% accuracy on this dataset, and the pattern of results across models and prompting styles is internally consistent. The use of SBERT to retrieve semantically similar examples and precomputed thumbnail descriptions is a sensible application of existing tools.\n\nThe soft spots are real but fixable. The biggest one is the comparison with CHECKER. The paper claims Claude 'matched or outperformed' CHECKER on every metric, but the two F1 scores come from different test sets: Claude's 0.7227 is on the 2,843-video ThumbnailTruth, while CHECKER's 0.7153 is on its own 197-video test set. Those numbers are not comparable, and the paper never runs either system on the other's data. That claim should be withdrawn or redone on a shared benchmark. The ablation text also contradicts Table 4: the full-input Claude zero-shot setting is described as best, but its accuracy (0.8920) is lower than ABL-NS and ABL-ND. That looks like a typo or misreading, but it needs correction.\n\nThe evaluation has a mild transductive circularity: dynamic few-shot examples are retrieved from the same dataset and carry ground-truth labels, and the best prompt is selected after seeing the full evaluation set. No confidence intervals or significance tests are reported. These issues do not sink the paper, but they mean the headline numbers should be treated as upper bounds, not as a fair estimate of out-of-distribution performance.\n\nThe annotation is what it is—two trained annotators' subjective judgments—but that is standard for this task and the paper is honest about it. The authors also acknowledge guardrail refusals and excluded videos rather than hiding them.\n\nWho is this for? Applied ML researchers and platform-governance folks who want a deployable baseline for thumbnail moderation. It deserves a serious referee, but the comparative claim and evaluation rigor need significant work before publication. My advice: send it to peer review with a request for major revision—keep the dataset, fix the CHECKER comparison, correct the ablation discussion, and add confidence intervals or a proper validation split.","headline":"A genuinely useful new dataset and a plausible LLM-prompting result, but the headline comparison with CHECKER does not survive contact with the actual numbers.","tokens_in":20143,"tokens_out":1913,"would_cite":false,"duration_ms":18673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using only prompts, Claude 3.5 Sonnet identifies misleading YouTube thumbnails with 93.8% accuracy across eight countries, matching or beating a supervised detector trained for the task.","keywords":["misleading thumbnails","YouTube","multimodal LLM","clickbait detection","dynamic few-shot prompting","content moderation","cross-cultural dataset","semantic alignment"],"falsifier":"Take a stratified sample of the 2,843 videos and have a larger panel of annotators from each of the eight countries label thumbnails as misleading without seeing the paper's codebook, then compare the panel labels with the paper's labels; substantial disagreement (say, more than 10% on any country) would show the 93.8% accuracy is an artifact of the original two annotators' shared biases rather than a measure of genuine deception.","tokens_in":19217,"feed_emoji":"🖼️","tokens_out":6207,"duration_ms":50017,"temperature":0.7,"pith_summary":"This paper tries to establish that a general-purpose multimodal LLM, prompted rather than trained for the task, can detect misleading YouTube thumbnails at levels competitive with specialised supervised systems. On a new dataset of 2,843 videos from eight countries—1,359 of them judged misleading, with over 7.6 billion collective views—the authors feed each model the thumbnail image, subtitle transcript, and an automatically generated video description, and ask it to judge semantic alignment. Claude 3.5 Sonnet with dynamic few-shot prompting reaches 93.8% accuracy, with precision above 92% and recall above 94% in several settings, and its F1 of 0.7227 edges out CHECKER's 0.7153, a supervised pipeline built for this exact task. Why it matters: if the result holds, platforms could screen thumbnails before upload without training bespoke detectors, and adapt to local clickbait styles by swapping in example videos rather than retraining.","feed_headline":"Claude 3.5 Sonnet hits 93.8% accuracy on misleading thumbnails","feed_subtitle":"A prompt-only LLM matches a trained detector's F1 without fine-tuning, and adapts to local thumbnail styles.","key_machinery":"The load-bearing object is the semantic-alignment judgment itself, assembled from three modalities per video: the thumbnail image, the subtitle transcript, and a model-generated scene-by-scene video description. The mechanism that sharpens it is dynamic few-shot prompting: Sentence-BERT embeddings of the video description retrieve one known-misleading and one known-honest example semantically closest to the test video, and those examples, with thumbnail descriptions and rationales, are inserted into the prompt. This turns the task into a comparison against local exemplars, which is what the authors credit for the cross-country accuracy gains, including at least 8 percentage points in Italy,","core_discovery":"The paper's central claim is that a general-purpose multimodal LLM, used with engineered prompts and no fine-tuning, can detect misleading YouTube thumbnails as accurately as a specialised supervised detector. Feeding the model three signals—the thumbnail image, the subtitle transcript, and an automatically generated scene-by-scene video description—and asking it to judge whether the thumbnail matches the content, Claude 3.5 Sonnet reaches 93.8% accuracy on a dataset of 2,843 videos from eight countries, with precision above 92%, specificity above 93%, and recall above 94% in several settings. Benchmarking against CHECKER, the best supervised pipeline for this task, the prompted Claude confi","pith_inferences":["Beyond the paper: the 93.8% figure is an agreement rate with a two-annotator codebook, not a measurement against viewer deception; a genuinely independent gold standard, such as post-viewing surveys per country, would likely lower the headline number, especially for culturally coded clickbait.","Beyond the paper: because the pipeline omits titles, comments, likes, and view counts by design, it cannot catch thumbnails that are deceptive only in relation to metadata or social signals; a hybrid system would be needed for post-hoc moderation.","Beyond the paper: dynamic few-shot retrieval depends on the quality of the generated video descriptions; if those descriptions contain errors, the retrieved examples can be off-topic, and the accuracy gain could shrink on videos with sparse or poor subtitles.","Beyond the paper: one could test the mechanism's limits by adversarial generation—feeding the pipeline thumbnails designed by an LLM to be subtly misleading—to see whether the semantic mismatch judgment degrades against adaptive creators."],"forward_implications":["Pre-upload screening becomes feasible: the pipeline uses only signals available before public release, so a misleading thumbnail can be flagged for human review at upload time rather than after viewers are exposed.","Cultural adaptation can be achieved by prompt selection, not retraining: swapping in locally relevant few-shot examples raised accuracy by at least 8 percentage points in Italy, Brazil, and Pakistan.","A general LLM can match a specialised supervised detector on this task: Claude's F1 of 0.7227 exceeds CHECKER's 0.7153, despite zero task-specific training.","High specificity (Claude above 93%) keeps false flags low enough that flagged thumbnails could be routed to human moderators without overwhelming them.","The same pipeline transfers across video categories: balanced per-category accuracy ranged from 0.875 for Science & Technology to 0.953 for Sports, suggesting the mechanism is not niche to one genre."],"supporting_citations":[{"why":"CHECKER supplies the supervised multimodal baseline that the paper's prompt-only Claude configuration must beat.","marker":"[4]"},{"why":"Sentence-BERT provides the text embeddings used to retrieve semantically similar examples for dynamic few-shot prompting.","marker":"[19]"},{"why":"Kojima et al. grounds the zero-shot chain-of-thought prompting strategy the paper builds on.","marker":"[17]"},{"why":"Twelve Labs generates the uniform video-to-text descriptions used as inputs to the classifiers and for semantic retrieval.","marker":"[13]"},{"why":"Anthropic's chain-of-thought documentation is invoked to explain Claude's structured reasoning advantage.","marker":"[30]"},{"why":"Zannettou et al. characterizes clickbait thumbnails as deviating substantially from content, informing the prompt's definition of misleading.","marker":"[18]"},{"why":"Hussein et al. supports the collection strategy that viewing misleading videos triggers recommendations of more misleading videos.","marker":"[6]"},{"why":"Lindstøl et al. supports the decision to exclude engagement signals, since users often scroll past deceptive content without reacting.","marker":"[7]"}],"fun_headline_variants":["Prompt-only LLM matches trained detector on misleading thumbnails","Claude 3.5 Sonnet: 93.8% accuracy on deceptive YouTube thumbnails","LLM flags misleading thumbnails across 8 countries, no fine-tuning","Deceptive thumbnails: zero-shot LLM rivals specialized detector","93.8% accurate: multimodal LLM catches thumbnail clickbait"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The paper treats the two annotators' agreement (Cohen's kappa 0.9633) as the ground truth for what makes a thumbnail misleading in all eight countries, so every accuracy figure is really a measure of agreement with that subjective codebook—if the codebook misses regional expectations, the headline 93.8% is not a true detection rate.","fun_headline_variants_meta":{"raw":{"variants":["Prompt-only LLM matches trained detector on misleading thumbnails","Claude 3.5 Sonnet: 93.8% accuracy on deceptive YouTube thumbnails","LLM flags misleading thumbnails across 8 countries, no fine-tuning","Deceptive thumbnails: zero-shot LLM rivals specialized detector","93.8% accurate: multimodal LLM catches thumbnail clickbait"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1723,"prompt_tokens":790,"completion_tokens":933,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":534,"tokens_out":933,"duration_ms":8605,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:56:50.073371+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a stratified sample of the 2,843 videos and have a larger panel of annotators from each of the eight countries label thumbnails as misleading without seeing the paper's codebook, then compare the panel labels with the paper's labels; substantial disagreement (say, more than 10% on any country) would show the 93.8% accuracy is an artifact of the original two annotators' shared biases rather than a measure of genuine deception.","supporting_citations":[{"cited_title":"Checker: Detecting clickbait thumbnails with weak supervision and co-teaching,","cited_arxiv_id":null,"evidence_quote":"CHECKER supplies the supervised multimodal baseline that the paper's prompt-only Claude configuration must beat."},{"cited_title":"Twelve labs: The platform for video understanding,","cited_arxiv_id":null,"evidence_quote":"Twelve Labs generates the uniform video-to-text descriptions used as inputs to the classifiers and for semantic retrieval."},{"cited_title":"Chain-of-thought prompting,","cited_arxiv_id":null,"evidence_quote":"Anthropic's chain-of-thought documentation is invoked to explain Claude's structured reasoning advantage."},{"cited_title":"The good, the bad and the bait: Detecting and characterizing clickbait on youtube,","cited_arxiv_id":null,"evidence_quote":"Zannettou et al. characterizes clickbait thumbnails as deviating substantially from content, informing the prompt's definition of misleading."},{"cited_title":"Measuring misinformation in video search platforms: An audit study on youtube,","cited_arxiv_id":null,"evidence_quote":"Hussein et al. supports the collection strategy that viewing misleading videos triggers recommendations of more misleading videos."},{"cited_title":"Adolescents’ experiences and (re)action towards fake news on social media: Perspectives from norway,","cited_arxiv_id":null,"evidence_quote":"Lindstøl et al. supports the decision to exclude engagement signals, since users often scroll past deceptive content without reacting."}],"review_version":1}