{"id":"4f44c071-f03d-4f37-80f5-e303ceb83c44","arxiv_id":"1909.01543","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multimodal classifier can flag likely misinformative prostate cancer videos on YouTube with roughly 74% accuracy, using a new expert-labeled dataset.","lead":"This paper introduces a doctor-labeled dataset of 250 YouTube videos about prostate cancer and trains machine learning models that tell trustworthy videos from misinformative ones. The best model reaches about 74% accuracy by combining spoken language, voice acoustics, and viewer engagement features.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy is likely inflated by query-specific vocabulary: misinformative videos were surfaced using 'miracle cure' and 'natural remedies' searches, so ngram features may memorize those lexical cues rather than detect misinformation generally.","rationale":"The reader's weakest assumption—that deliberate keyword-based sampling limits generalization to the full distribution of YouTube prostate cancer videos—is on target, and this stress-test sharpens it into a specific mechanism: contamination between the retrieval queries and the ngram feature space. The central claim is conditional on the classifier learning misinformation cues rather than query lexicon. The proposed leave-one-query-out test directly probes this by withholding one trigger-query group and checking whether the model still detects misinformation expressed with different vocabulary. I do not think this rises to rejection because the paper is transparent about several limitations and the dataset/annotation effort is a real contribution; however, the search-term confound deserves an explicit experiment before the results can be accepted as generalizable. The reader's CONDITIONAL verdict remains appropriate, hence UNCHANGED. Other possible issues, such as the arithmetic discrepancy between the abstract's '55% error rate reduction' and the approximately 46% computed from Table 3, are real but do not undermine the reported accuracy itself; they are secondary to the external-validity concern.","tokens_in":11614,"tokens_out":4694,"duration_ms":53189,"concrete_test":"Conduct a leave-one-query-out cross-validation: group the videos by the YouTube search query that retrieved them (e.g., 'prostate cancer miracle cure' vs 'prostate cancer natural remedies'), train the (1)+Ling+(8) SVM on all query groups except one, and evaluate misinformative-class precision and recall on the held-out query group. If recall on the held-out query falls materially below the reported 73.15% in Table 3, the classifier is exploiting retrieval-specific vocabulary rather than a general misinformation signal, confirming the sampling-bias concern and requiring the claim to be re-scoped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—74.41% accuracy for the combined model in Table 3—requires that the learned decision boundary generalizes beyond the retrieval protocol. Section 3 (Data Collection) describes collecting trustworthy videos with queries like 'prostate cancer screening' and misinformative videos with trigger queries like 'prostate cancer miracle cure' and 'prostate cancer natural remedies'. The best model includes 3,577 ngram features (unigrams and bigrams with tf-idf weighting), so surface terms from the trigger queries are prime discriminators. Table 2's own misinformative transcript repeats 'miracle cure' multiple times, and the annotation rubric treats anecdotal 'cure' claims as misinformative, confounding the label with the search vocabulary. The model may thus achieve high accuracy by recognizing query-specific phrasing instead of general misinformation cues. Section 5 acknowledges exclusion-related biases (animations, voice-overs) and overrepresentation of laypeople, but it does not address this search-term confound. If the ngram signal is dominated by retrieval lexicon, the claimed 46% error-rate reduction over the majority baseline is partly an artifact of sampling, and the method would not transfer to naturally occurring prostate cancer videos that do not use these phrases. This external-validity gap is the load-bearing weakness in the paper's central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses automatic detection of misinformation in YouTube videos about prostate cancer. The authors introduce a new dataset of 250 videos manually annotated on a 5-point misinformation scale and binarized into trustworthy and misinformative classes. They extract viewer engagement, linguistic (ngrams, LIWC, syntax, readability, lexical richness), and acoustic (openEAR) feature sets, and train linear SVM classifiers with five-fold cross-validation. The best combined model is reported to reach 74.41% accuracy, with 76.51% precision and 73.15% recall for the misinformative class, against a 52.8% majority-class baseline. The paper also reports ablation results showing that individual feature sets vary widely and that the combined model outperforms single-modality models.","tokens_in":11869,"tokens_out":3192,"duration_ms":32100,"significance":"If the reported accuracy transfers beyond the specific retrieval protocol, this is a useful early step toward automated screening of health misinformation in online video, a largely underexplored problem. The main strengths are the new expert-annotated dataset, the explicit majority-class baseline, and the systematic ablation across multimodal feature groups. The central risk is external validity: because misinformative videos were retrieved using trigger phrases such as 'miracle cure' and 'natural remedies' while trustworthy videos were retrieved using neutral medical queries, the high ngram-based accuracy may reflect query-specific vocabulary rather than generalizable misinformation cues. The paper's contribution is therefore conditional on additional robustness evidence showing that the model generalizes beyond its search-term vocabulary.","major_comments":[{"comment":"","section":"Section 3 (Data Collection) and Table 3"},{"comment":"","section":"Table 3 and Section 4 (Classification Results)"},{"comment":"","section":"Section 3 (Data Annotation)"}],"minor_comments":[{"comment":"","section":"Abstract and Section 1"},{"comment":"","section":"Table 3"},{"comment":"","section":"Section 4 (Viewer Engagement Features)"},{"comment":"","section":"Section 3 (Privacy Considerations)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for ICMI and addresses a timely problem, but the search-term confound is the key risk to the central claim. I would support acceptance after the authors add a negative-control experiment that separates query vocabulary from general misinformation cues, together with confidence intervals or nested cross-validation. The current manuscript is a solid systems paper whose conclusions outrun the evidence on external validity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is a new, expert-labeled dataset of 250 prostate cancer YouTube videos and a careful ablation study showing a multimodal SVM reaching 74% accuracy. That is a legitimate first benchmark for an under-served problem. But the headline number is weaker than it looks because of how the data was collected: misinformative videos were deliberately surfaced with searches like 'prostate cancer miracle cure' and 'prostate cancer natural remedies', so the n-gram features are likely learning the retrieval lexicon rather than general misinformation cues.\n\nWhat the paper does well: the annotation by urologists with high within-one agreement, the sensible feature groupings (engagement, linguistic, acoustic), and the clear ablations. The finding that user engagement alone is weak but adds something in fusion is plausible. The authors are also honest about several limitations, including the bias toward laypeople and the exclusion of animation/voice-over content. Credit where due.\n\nThe soft spots, in order of importance. First, the sampling confound is load-bearing. The annotation rubric counts anecdotal 'cure' claims as misinformative, and the example transcript in Table 2 repeats 'miracle cure' several times. A bag-of-ngrams model will lock onto those surface cues. Five-fold CV measures performance within the distribution created by this retrieval protocol; it says little about transfer to naturally occurring prostate cancer videos that don't use those phrases. So the 74% accuracy and the claimed 'error rate reduction' are likely optimistic for the real-world task. The paper's limitations section does not address this confound. Second, the abstract says the error-rate reduction is 55%, but from the reported numbers the actual reduction is about 46% (from 47.2% to 25.6% error). Sloppy but fixable. Third, the dataset and code are not released, which limits reproducibility; a benchmark that cannot be downloaded is less useful.\n\nThe core argument—that multimodal features separate the two classes within this sample—holds up. The broader claim that the model detects misinformation in the wild is not established.\n\nThis paper is for researchers working on health misinformation in video, and for anyone teaching dataset construction pitfalls. It deserves a serious referee; I'd send it out and ask for major revision: address the confound, either by collecting a validation set of naturally occurring videos or by explicitly showing the model still works when those query-specific n-grams are removed, correct the error-rate figure, and release the data. I would not cite it as a clean benchmark without those fixes, but the dataset is worth knowing about.","headline":"A genuinely new dataset with a headline accuracy that is likely inflated by the query terms used to collect misinformative videos; deserves peer review with major revision.","tokens_in":12383,"tokens_out":3190,"would_cite":false,"duration_ms":32675,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multimodal SVM can identify misinformative prostate cancer YouTube videos with 74% accuracy.","keywords":["misinformation detection","prostate cancer","YouTube","multimodal classification","support vector machine","health misinformation","acoustic features","linguistic features"],"falsifier":"Collect a fresh, unfiltered sample of prostate cancer videos from YouTube, including animations, voice-overs, non-English content, and videos longer than 30 minutes, annotate it with the same expert protocol, and run the published feature pipeline with the same SVM settings. If accuracy on that sample drops toward the 52.8% majority baseline, the original 74% result is an artifact of the curated search and filtering procedure.","tokens_in":11392,"feed_emoji":"🩺","tokens_out":6821,"duration_ms":65760,"temperature":0.7,"pith_summary":"This paper asks whether a computer can tell when a YouTube video about prostate cancer is spreading misinformation, without a doctor watching it. It introduces a new annotated dataset of 250 prostate cancer videos—132 trustworthy and 118 misinformative—and shows that a linear SVM combining viewer engagement statistics, transcript language features, and acoustic-prosodic features identifies misinformative videos with 74.4% accuracy, 76.5% precision, and 73.2% recall. That is a clear improvement over the 52.8% majority-class baseline, cutting the error rate roughly in half. If the result holds, it means cheap automated screening could flag suspicious health videos for expert review at scale.","feed_headline":"One classifier flags misleading prostate cancer videos at 74%","feed_subtitle":"Words, tone of voice, and viewer engagement together beat the majority baseline by a wide margin.","key_machinery":"The load-bearing mechanism is an early-fusion linear SVM over three feature families: six YouTube engagement statistics (views per day, comments, thumbs up and down, duration, and category), 6,988 linguistic features computed from automatically transcribed and punctuation-restored transcripts, and 384 Emo_IS09 acoustic features. Labels come from expert urologist ratings on a five-point misinformation scale collapsed into a binary trustworthy/misinformative split. The ablation design—each feature family alone, then combined—is what lets the paper attribute the final accuracy to complementarity across modalities.","core_discovery":"On the paper's own terms, the central discovery is that misinformation in medical videos is detectable from surface signals: a classifier that fuses all extracted linguistic features, six YouTube engagement attributes, and the Emo_IS09 acoustic feature set reaches 74.41% accuracy, with 76.51% precision and 73.15% recall on the misinformative class. The integrated model outperforms every single-modality model, and ngrams are the strongest individual linguistic feature, followed by syntax features and LIWC categories. The paper also establishes through ablation that the acoustic and engagement channels add complementary signal on top of language rather than simply duplicating it.","pith_inferences":["A natural next test is cross-topic transfer: because many ngram features are prostate-cancer-specific, the same pipeline may need topic-adapted features to detect misinformation about other diseases.","The time-sensitive engagement features mean a deployed model trained on April 2019 statistics would age; a practical detector would need periodic retraining or should drop engagement features.","If acoustic cues truly help, a targeted experiment could measure which acoustic dimensions—pitch range, loudness, speaking rate—separate anecdotal 'miracle cure' testimonials from clinical explanations.","The declared filtering criteria imply the strong accuracy may be partly a curated-sample effect; the strongest testable extension is running the same pipeline on unfiltered YouTube search results."],"forward_implications":["A first-pass automated screener could rank YouTube prostate cancer videos by likelihood of misinformation and send only flagged or uncertain videos to medical experts, lowering the cost of manual review.","Viewer engagement alone is not enough: it reaches 96% precision but only 21% recall on misinformative videos, so a deployed detector needs the linguistic and acoustic channels to avoid missing most bad content.","The combined model's improvement over language-only features shows that tone of voice and audience reaction carry independent misinformation cues that transcripts alone miss.","The annotated 250-video dataset, with fine-grained scores from 1 to 5, can support future work on graded misinformation severity instead of only binary flags."],"supporting_citations":[{"why":"Found a negative correlation between scientific quality and viewer engagement in prostate cancer YouTube videos, motivating both the dataset and the engagement features.","marker":"[23]"},{"why":"Physician ratings of 51 prostate cancer YouTube videos established that most were fair or poor quality, motivating automatic screening.","marker":"[33]"},{"why":"Previous automatic medical-knowledge classification of YouTube videos with bidirectional LSTMs is the closest prior automatic method this work extends.","marker":"[20]"},{"why":"Showed that readability, tentative language, and avoidance of personal pronouns mark cancer misinformation on Twitter, motivating LIWC and readability features.","marker":"[14]"},{"why":"DISCERN quality criteria shaped the five-point misinformation annotation scale used to label the videos.","marker":"[6]"},{"why":"Documented misinformation prevalence in psoriasis YouTube videos, supporting the engagement-based view of misinformation spread.","marker":"[29]"},{"why":"Provided the raw acoustic feature extraction toolkit used to generate the Emo_IS09, emobase, and emo_large feature sets.","marker":"[10]"},{"why":"Supplied the linear SVM implementation used in all classification experiments.","marker":"[27]"}],"fun_headline_variants":["AI flags misleading prostate cancer videos at 74% accuracy","Detect medical video misinformation with 74% accuracy","New model spots unreliable medical videos 74% of the time","Misinformation in medical videos detected with 74% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset was built by searching for obviously misinformative phrases like 'miracle cure' and by excluding animated videos, artificial voices, non-English speech, more than two speakers, and videos over 30 minutes, so the learned boundary may not hold for the full population of prostate cancer videos on YouTube.","fun_headline_variants_meta":{"raw":{"variants":["AI flags misleading prostate cancer videos at 74% accuracy","Detect medical video misinformation with 74% accuracy","New model spots unreliable medical videos 74% of the time","Misinformation in medical videos detected with 74% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2737,"prompt_tokens":848,"completion_tokens":1889,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1822}},"tokens_in":464,"tokens_out":1889,"duration_ms":14313,"temperature":1.0,"reasoning_tokens":1822,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:13:43.295496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a fresh, unfiltered sample of prostate cancer videos from YouTube, including animations, voice-overs, non-English content, and videos longer than 30 minutes, annotate it with the same expert protocol, and run the published feature pipeline with the same SVM settings. If accuracy on that sample drops toward the 52.8% majority baseline, the original 74% result is an artifact of the curated search and filtering procedure.","supporting_citations":[{"cited_title":"Macaluso, Stefan W","cited_arxiv_id":null,"evidence_quote":"Found a negative correlation between scientific quality and viewer engagement in prostate cancer YouTube videos, motivating both the dataset and the engagement features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Physician ratings of 51 prostate cancer YouTube videos established that most were fair or poor quality, motivating automatic screening."},{"cited_title":"YouTube for Patient Education: A Deep Learning Approach for Understanding Medical Knowledge from User-Generated Videos","cited_arxiv_id":"1807.03179","evidence_quote":"Previous automatic medical-knowledge classification of YouTube videos with bidirectional LSTMs is the closest prior automatic method this work extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Showed that readability, tentative language, and avoidance of personal pronouns mark cancer misinformation on Twitter, motivating LIWC and readability features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DISCERN quality criteria shaped the five-point misinformation annotation scale used to label the videos."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documented misinformation prevalence in psoriasis YouTube videos, supporting the engagement-based view of misinformation spread."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provided the raw acoustic feature extraction toolkit used to generate the Emo_IS09, emobase, and emo_large feature sets."}],"review_version":1}