{"id":"91860660-1b34-4437-84fb-47d0653c3d36","arxiv_id":"2411.09577","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SimTube generates pre-publication video comments from multimodal video understanding and sampled user personas, and its evaluations suggest these simulated comments are often rated as helpful as real ones.","lead":"The paper introduces SimTube, a system that generates simulated YouTube-style comments for a video before it is published, using AI models that read the video's visuals, audio, and metadata, plus user personas. It could help creators get early, varied feedback instead of waiting for real audiences to comment after release.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim rests on a biased baseline: real comments are sampled from the top-1000 popular comments (Sec 6.1.1), which the paper itself calls brief and laudatory (Sec 6.1.3); if the baseline broadened, the 'more informative' margin may vanish.","rationale":"The reader's weakest assumption is exactly the load-bearing concern. The paper explicitly flags the baseline problem in Sec 6.1.3, and the abstract's most striking comparative claim rests on this evidence. The proposed test would settle whether the margin survives a fairer baseline. The system itself is coherent and the qualitative user study is positive, but those results support SimTube as a promising pre-publication feedback tool, not the strong claim that it beats actual audience feedback. Therefore the existing conditional verdict remains appropriate; the concern does not change the verdict but does define the condition that must be addressed.","tokens_in":21495,"tokens_out":3936,"duration_ms":41271,"concrete_test":"Re-run the crowd-sourced evaluation on the same eight videos using three baselines: (i) a random sample from the full comment stream (not restricted to the top 1000), (ii) a top-1000 sample length-controlled by truncating generated comments to match real-comment length, and (iii) a sample of all real comments longer than 100 characters. Compare Helpfulness and Informativeness ratings across conditions. If SimTube's margin over baseline (i) or (ii) is no longer statistically significant, the abstract's 'more informative than actual audience comments' should be revised to 'more informative than popular short comments.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SimTube comments are 'often more detailed and informative than actual audience comments' depends on the comparison set used as 'actual audience comments.' In the crowd-sourced study (Sec 6.1.1), real comments were randomly sampled from the top 1000 most popular comments per video, ranked by YouTube meta-ratings. The authors themselves observe (Sec 6.1.3) that these popular comments 'are often brief and offer limited informative value for creators' and 'sometimes purely laudatory.' This is not a neutral baseline: it selects for short, humorous, or approval-type comments, while excluding the longer, more critical, or question-based comments that creators actually receive in the full comment stream. The Appendix G examples illustrate the asymmetry: real comments such as 'Is it bird? Is it a plane? Nah... it's a piece of garlic bread :D' are rated against generated comments that explicitly include constructive suggestions. Because the headline claim is comparative ('more detailed and informative'), it is load-bearing that the baseline represent actual audience feedback; the current baseline is curated away from the attributes being compared. If the baseline were instead a random sample of all comments, or comments matched for length and content, the reported advantage on Helpfulness and Informativeness could shrink or disappear. This does not invalidate SimTube as an ideation tool, but it undercuts the strong comparative claim in the abstract.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SimTube, a full-stack system that generates simulated YouTube comments from videos by combining multimodal video understanding (Whisper transcription, LLaVA-NeXT frame captioning, LLM summarization) with persona-based prompting. The user interface lets creators upload videos, inspect generated comments, expand threads, and craft custom personas. The evaluation includes a crowd-sourced study (25 Upwork raters) comparing relevance, believability, and helpfulness of SimTube-generated comments against actual YouTube comments; automatic diversity and relevance metrics (Distinct N-gram, Self-BLEU, BERTScore, ROUGE, LLM Eval); and a qualitative user study with eight experienced creators. The paper claims that SimTube's generated comments are not only relevant, believable, and diverse but often more detailed and informative than actual audience comments.","tokens_in":21808,"tokens_out":4083,"duration_ms":36759,"significance":"If the central claim holds, SimTube offers video creators a scalable way to obtain audience-like feedback before publication, which could benefit content iteration and reduce reliance on post-publication comments. The paper's strengths include a complete implemented system, a multi-part evaluation that includes independent human crowd ratings, and a qualitative user study with experienced creators that surfaces concrete workflow integration scenarios. The crowd-sourced ratings are an external grounding for the relevance and believability of the generated comments. However, the headline comparative claim depends on a real-comment baseline that the authors themselves characterize as brief and laudatory, and the automatic relevance metrics are partly circular because the generated comments are produced from the same video summary used as the evaluation reference. These issues must be resolved before the comparative claim is fully established.","major_comments":[{"comment":"The real-comment baseline is not a neutral sample of audience feedback: Section 6.1.1 states that comments were randomly sampled from the top 1000 popular comments per video, ranked by YouTube meta-ratings, and Section 6.1.3 admits that these popular comments 'are often brief and offer limited informative value for creators' and are 'sometimes purely laudatory.' Because the headline claim in the abstract is explicitly comparative ('more detailed and informative than actual audience comments'), this curated baseline biases the comparison in favor of SimTube. The crowd-sourced study should be repeated with a random sample of all comments (after filtering), or with comments matched for length and informativeness, to support the claimed advantage.","section":"Sec. 6.1.1 and Sec. 6.1.3"},{"comment":"The automatic relevance metrics (ROUGE, BERTScore, LLM Eval) compare generated comments against the video summary and keywords that the pipeline itself generates in Section 5.1.3. Since the comment generation prompt in Section 5.3.1 is given this same summary and keywords, high word-level and semantic overlap with that summary is partly by construction, making the reported relevance advantage over real comments unsurprising. The independent crowd-sourced relevance ratings provide stronger evidence, but the automatic metrics should be reframed as measuring consistency with the system's own summary rather than as independent evidence of relevance.","section":"Sec. 6.2.2 and Sec. 5.1.3"},{"comment":"The abstract claims that user personas are 'derived from a broad and diverse corpus of audience demographics,' but the actual persona source is PersonaChat, a general persona dataset from a dialogue task, not a corpus of YouTube audience demographics. Furthermore, Section 6.1.3 reports that the full system with personas did not significantly outperform the No-persona ablation, and Section 7.3.2 reports mixed user reactions to persona-based comments. The paper's contribution claim for the persona module is therefore not supported by the presented evidence; either the wording should be corrected or additional analyses (e.g., measuring persona diversity or demonstrating a significant benefit) should be provided.","section":"Sec. 5.2.1 and Abstract"}],"minor_comments":[{"comment":"The sentence 'For each video, we sampled 30 comments from three categories' is ambiguous: it is not clear whether 30 comments are sampled per category (90 total per video) or 30 comments total across all three categories. This should be clarified because it determines the number of ratings collected.","section":"Sec. 6.1.1"},{"comment":"The statement that 'ground-truth video summaries were compiled from 30 User Study participants using Claude 3.5 Sonnet' is unclear: it should specify how the participant-provided summaries were aggregated or processed by the LLM, and whether the same model was used in the LLM evaluation, which could introduce a self-preference bias.","section":"Sec. 6.2.2"},{"comment":"The claim that the frame-panel approach is 'a unique approach' is an overstatement; paneling multiple frames for VLM input has been explored in prior video understanding work. I recommend softening this wording.","section":"Sec. 5.1.2"},{"comment":"There is a typo: 'calcualte' should be 'calculate.' Similar typos appear in Appendix C ('isscooping') and Section 4.3 ('comment' missing a noun).","section":"Sec. 5.2.2"},{"comment":"The description of the Self-BLEU plot in Section 6.2.1 is confusing: the text refers to 'the right side of the plot titled Self-BLEU' and then mentions 'the left half' without clear labels in the figure. Please make the figure and caption self-explanatory.","section":"Figure 5"},{"comment":"Section 5.3 states the system directly generates 30 comments with 70% primary and 30% thread comments, but Section 6.1.1 also mentions 30 comments per video for evaluation; please clarify whether these are the same 30 comments and how they relate to the 671 and 1141 comments used in the automatic metrics.","section":"Sec. 5.3"}],"recommendation":"major_revision","confidential_remarks":"This is a well-engineered system paper with a credible multi-part evaluation, but the comparative claim in the abstract is not yet supported by the current baseline. The authors may be able to fix this by re-running the crowd study with a more representative real-comment sample or by rewriting the claim to acknowledge that the advantage is over popular, short comments. The automatic metric circularity is a second concern that should be addressed by a clearer framing. I would not recommend rejection, as the system itself appears useful and the evaluation scope is broad; however, the revisions are substantive rather than purely cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SimTube is a real system paper. The pipeline—Whisper transcription, LLaVA-NeXT frame captioning with frame-dialogue panels, LLM summarization, persona retrieval from PersonaChat, then comment generation with persona prompting—is coherent and described in enough detail to reimplement. The evaluation is genuinely multi-part: crowd-sourced ratings from 25 Upworkers, automatic metrics (including LLM judges from two different model families), and a qualitative study with eight creators. The UI ideas, especially thread expansion and persona crafting, are sensible, and the interview excerpts give concrete evidence that creators find the simulated comments useful for rough-cut feedback. That is a legitimate contribution to creator tools.\n\nThe soft spot is exactly where the stress-test points. The abstract claims generated comments are 'often more detailed and informative than actual audience comments.' That claim is load-bearing, and the baseline is not a fair sample of actual comments: real comments are drawn from the top-1000 most popular per video, and the paper itself notes in Section 6.1.3 that these 'are often brief and offer limited informative value for creators' and sometimes purely laudatory. Of course longer, deliberately instructive generated comments beat brief one-liners on 'helpfulness.' Broaden the baseline to a random sample of all comments, or match for length and content, and the margin plausibly shrinks or disappears. This does not invalidate SimTube as an ideation tool, but it means the headline comparison is overstated. The authors are honest enough to flag the nature of the baseline, but they still lean on the comparative claim.\n\nOther gaps are smaller. No code or data are released, which hurts reproducibility. There are no baselines against prior generative comment systems (LiveBot, PLVCG), so the novelty claim is not tested against the closest prior work. The user study is eight self-selected creators—fine for qualitative insight, not for generalizing. And calling PersonaChat a 'broad and diverse corpus of audience demographics' is a stretch; it is 8,000 personas, not a YouTube-audience sample.\n\nOn the plus side, they used two different LLMs for relevance evaluation to mitigate self-generation bias, and they discuss the limitations of their own approach (ignoring editing techniques, genre variation) in the discussion. The thinking is honest and the system is a solid engineering contribution.\n\nVerdict: send it out. A serious referee will want the baseline fixed or the claim softened, but the system and evaluation are worth engaging with. I'd bring it to reading group and would probably cite it for the system design and the user-study findings.","headline":"A solid system paper whose usefulness survives the weak baseline, but the abstract's comparative claim needs to be walked back or re-tested on a fair sample of real comments.","tokens_in":22350,"tokens_out":2636,"would_cite":true,"duration_ms":26001,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SimTube generates simulated audience comments before a video is published, and its comments are rated more relevant, believable, and helpful than actual popular comments.","keywords":["simulated video comments","audience feedback","multimodal video understanding","user personas","large language models","comment generation","video content creation"],"falsifier":"Take the same eight videos and ask blind raters to evaluate generated comments against real comments sampled from the full comment distribution, or matched for length, and check whether generated comments still score higher on helpfulness. A second check is to ask experienced creators to identify which comments were AI-generated; if they can reliably distinguish them, the believability claim weakens.","tokens_in":21291,"feed_emoji":"💬","tokens_out":6291,"duration_ms":54132,"temperature":0.7,"pith_summary":"The paper introduces SimTube, a generative system that produces simulated audience comments for videos that have not yet been published. Its goal is to let creators obtain diverse, believable feedback before release, instead of waiting for real viewers to comment. The authors' central claim is that SimTube's comments are not only relevant, believable, and diverse but often more detailed and informative than actual audience comments. If that holds, creators could use pre-publication simulated feedback to refine their current video, and novice creators with small audiences would gain a scalable source of input.","feed_headline":"Simulated comments outscore popular ones on relevance and helpfulness","feed_subtitle":"A multimodal pipeline with reader personas turns rough cuts into pre-publication audience feedback for creators.","key_machinery":"The load-bearing mechanism is a three-stage backend pipeline. First, Video Understanding transcribes the audio with Whisper, captions one-second segments by giving a vision-language model (LLaVA-NeXT 13B) a four-frame 'panel' paired with the dialogue, and then summarizes the captions and transcript into a video summary and keywords using a long-context LLM (Claude 1.6). Second, Persona Query embeds the keywords and over 8,000 PersonaChat personas, ranks them by cosine similarity, and keeps the top 30 relevant personas. Third, Comment Generation prompts an LLM with few-shot examples of real video comments, the video metadata, summary, keywords, and personas to produce primary comments and thread replies, and it also generates comments from user-defined personas and replies in threads. The UI lets creators expand threads and craft their own personas.","core_discovery":"On the paper's own terms, SimTube shows that multimodal video understanding plus persona-based generation can produce comments that match or exceed real video comments in relevance, believability, and helpfulness. In a crowd-sourced study with 25 workers rating comments on eight videos, both the full system and a no-persona ablated version scored significantly higher than real comments on relevance and believability, and the full system also scored significantly higher on helpfulness; the difference between the full and no-persona versions was not statistically significant. Automatic metrics showed generated comments have higher word-level diversity and stronger word- and semantic-level relevance to the video, while real comments retain greater semantic diversity. The paper frames this as evidence that simulated comments can supply preliminary inspiration and actionable feedback before publication, complementing rather than replacing real human comments.","pith_inferences":["Editorial extension: the same pipeline could be applied to thumbnails, titles, or rough-cut segments to predict audience reactions beyond comments.","Testable extension: comparing generated comments against a length-matched or full-distribution sample of real comments would show whether the helpfulness advantage persists when the baseline is not dominated by brief, laudatory popular comments.","Potential risk: if creators rely heavily on simulated feedback, they may overfit to a generic, polite audience and miss the unconventional reactions that drive real engagement; the paper itself recommends simulated comments as a supplement, not a replacement."],"forward_implications":["Creators can gather audience-style feedback before publishing, potentially improving the current video rather than only future episodes.","Novice creators with few real viewers gain a scalable source of simulated input that does not depend on posting first.","Persona crafting and thread expansion let creators ask what their target audience would think and follow up with specific questions.","Because the no-persona version performed nearly as well as the full system on the headline metrics, simpler pipelines may already provide useful feedback.","Simulated comments are typically more constructive and less harsh than real comments, which may reduce emotional harm during review."],"supporting_citations":[{"why":"Supplies Whisper, the speech-recognition model that transcribes video dialogue and narration for the video-understanding stage.","marker":"[58]"},{"why":"Provides the PersonaChat dataset of over 8,000 personas that the system queries for diverse comment generation.","marker":"[47]"},{"why":"Supplies the LLM-Eval method the paper adopts to rate comment relevance on a 0-100 scale.","marker":"[42]"},{"why":"Supplies BERTScore, used to measure semantic relevance between comments and video summaries.","marker":"[85]"},{"why":"Supplies ROUGE, used to measure word-level relevance of comments to video content.","marker":"[41]"},{"why":"Supports the use of meta-ratings to filter the real-comment baseline to popular, non-offensive comments.","marker":"[62]"}],"fun_headline_variants":["Simulated comments top real ones on relevance and believability","AI comments match human feedback in pre-release video tests","Simulated comments outscore actual audience feedback in study","Pre-release AI comments rival real ones for video creators","SimTube comments as informative as real audience feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that real comments sampled from the top 1000 popular comments on each video fairly represent genuine audience feedback; the paper itself notes these comments are often brief and offer limited informative value for creators, so the comparison may exaggerate SimTube's advantage.","fun_headline_variants_meta":{"raw":{"variants":["Simulated comments top real ones on relevance and believability","AI comments match human feedback in pre-release video tests","Simulated comments outscore actual audience feedback in study","Pre-release AI comments rival real ones for video creators","SimTube comments as informative as real audience feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001206,"raw_usage":{"total_tokens":4920,"prompt_tokens":851,"completion_tokens":4069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":3992}},"tokens_in":467,"tokens_out":4069,"duration_ms":27430,"temperature":1.0,"reasoning_tokens":3992,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:30:01.949392+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same eight videos and ask blind raters to evaluate generated comments against real comments sampled from the full comment distribution, or matched for length, and check whether generated comments still score higher on helpfulness. A second check is to ask experienced creators to identify which comments were AI-generated; if they can reliably distinguish them, the believability claim weakens.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PersonaChat dataset of over 8,000 personas that the system queries for diverse comment generation."}],"review_version":1}