{"id":"98ef0c3d-b329-4904-9739-ba499b9d1680","arxiv_id":"1908.03451","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A similarity-based neural network with word-level attention and noise-weighting detects spoiler comments in time-sync video comments, reporting an average 11.2% F1 improvement over prior methods.","lead":"This paper builds a neural network that detects spoiler comments posted in sync with online videos by comparing each comment's similarity to its neighbors and to key dramatic moments. The model reportedly beats earlier spoiler detectors by about 11% in F1 score on Chinese video comment data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Keyword pre-filtering in Section 5.1 biases the ground truth: unexamined TSCs are labeled non-spoilers, so the reported 11.2% F1 gain may reflect keyword memorization rather than genuine spoiler detection.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: keyword-based candidate selection corrupts the ground truth. This is the most consequential issue because the paper's headline result is a comparative F1 improvement; if the labels are biased in the way described, the comparison is not measuring what it claims. I do not find an additional, more severe weakness. The loss-function sign error and the R/P inconsistency are real but secondary: they are easily corrected in a revision and do not directly undermine the empirical comparison unless reimplementation fails. The proposed model itself is coherent and the paper provides enough architectural detail to be implemented, but no code or data is released, so the label-bias concern cannot currently be checked. A conditional acceptance is therefore the right stance: the authors should be asked to either release the annotation pipeline and data, or re-run evaluation on an independently and unbiasedly annotated sample. If that test confirms the F1 gap, the paper's central claim would be supported; if not, the claim should be revised. My read does not change the reader's verdict, so I mark verdict_should_be as UNCHANGED.","tokens_in":14551,"tokens_out":2589,"duration_ms":28474,"concrete_test":"Re-annotate a random sample of the original TSC corpora without any keyword pre-filtering: have the three evaluators apply the same 'related to later plot' criterion to every sampled TSC, then measure spoiler prevalence and evaluate the trained SBN-IVA plus the strongest baseline (DN-GAA) on this unbiased sample. If the F1 gap shrinks substantially or if the model's precision on keyword-free spoilers is near chance, the reported 11.2% improvement is an artifact of the biased labeling scheme.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (11.2% average F1 improvement over DN-GAA) rests on the reliability of the spoiler labels. Section 5.1 states: 'we first summarize the possible spoiler keywords specific to each video’s tags and filter out the TSCs that most likely to be spoilers based on keywords. Then three human evaluators check the high-likely spoilers.' This means only TSCs that match a preselected keyword list are ever reviewed; all other TSCs are automatically treated as non-spoilers. Consequently, a genuine spoiler that does not contain any preselected keyword is silently labeled as non-spoiler. This biases both training and test distributions: positives are drawn only from the keyword-matched candidate pool, while negatives conflate true non-spoilers with unexamined keyword-free spoilers. A model can therefore achieve high F1 by detecting the same shallow keyword cues used to build the labels, and the reported 11.2% advantage over the state-of-the-art baseline may reflect this label leakage rather than the model's interactive/time-decay/IVA design. The unusually high spoiler proportions in Table 2 (TV-series 0.3044; movies 0.1689; sports 0.1670) are consistent with positive candidates being pre-selected by keywords. Since the same biased labels are used in the test set, the absolute numbers and the relative ranking over baselines are both called into question. Secondary correctness issues (Eq. 14 missing the negative sign for binary cross-entropy; inconsistent R/P settings between Section 5.2 and the discussion after Table 4) do not by themselves decide the central claim, but they reinforce the need for external validation of the experimental pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SBN-IVA, a deep neural network for detecting spoiler time-sync comments (TSCs) on Chinese video platforms. The model encodes each TSC with a word-level attentive Bi-LSTM, computes semantic similarity between the target TSC and both its temporally preceding neighbors (with time-decay weighting) and a set of keyframes, and uses a sentence-level IVA module to down-weight noise comments. The final spoiler score is the sigmoid of the difference between maximum keyframe similarity and weighted neighbor similarity. The authors evaluate on self-collected Youku/Bilibili datasets for TV-series, movies, and sports, reporting F1 scores of 0.850, 0.811, and 0.825 respectively, exceeding several baselines including DN-GAA by an average of 11.2% in F1.","tokens_in":14900,"tokens_out":5872,"duration_ms":53078,"significance":"If the reported results are reliable, the work is a useful domain-specific contribution to spoiler detection, combining textual attention, temporal interaction, and keyframe similarity in a novel way, and the IVA mechanism for noise reduction is an interesting idea. The paper also presents a concrete architecture and an extensive dataset (over 1.5M TSCs). However, the evaluation is undermined because the ground-truth labels are constructed by keyword pre-filtering with human review of keyword-matched comments only, and because the keyframe-selection rule is justified from the same keyword-derived labels. These issues prevent the central performance claim from being accepted at face value. The paper would be strengthened by additional human annotation of a random sample, significance testing, and correction of the noted inconsistencies.","major_comments":[{"comment":"The labeling procedure first filters TSCs by pre-defined spoiler keywords and only has humans check those high-likely spoilers; all other TSCs are automatically labeled non-spoilers. This means any spoiler that does not contain a preselected keyword is never reviewed and is mislabeled as a negative. Since the same labels are used for training and testing, the reported F1 improvements and the 11.2% average gain may reflect the model's sensitivity to the same keyword cues used to build the labels rather than to genuine spoiler content. I ask the authors to quantify the recall of the keyword filter on a human-annotated random sample (including TSCs not containing keywords) and to re-run the comparison on a test set where every TSC is human-annotated.","section":"Section 5.1"},{"comment":"The keyframe selection rule (last quarter of the video, 10-second windows with highest TSC density) is justified by the cumulative occurrence proportion of \"key-TSCs,\" which are defined as TSCs containing the same spoiler keywords used for labeling. This makes the keyframe feature and the ground truth jointly determined by the same keyword heuristic, creating a circularity that can inflate keyframe-similarity's contribution. The authors should validate keyframes against independently human-annotated plot-revealing moments, or at least show that keyframe selection is robust to the keyword list.","section":"Section 3.1 and Fig. 2"},{"comment":"The experiments are repeated 10 times but only point estimates of Precision, Recall, and F1 are reported, without standard deviations, confidence intervals, or significance tests. Given the small margins between some conditions (e.g., SBN-IVA vs SBN on movies, F1 0.811 vs 0.785), the claimed improvements cannot be assessed. Add error bars and paired statistical tests (e.g., per-video bootstrap or paired t-test) for the comparisons in Table 3.","section":"Table 3"},{"comment":"The binary cross-entropy loss is written without the negative sign and without averaging over samples. As written, minimizing L = y ln ŷ + (1-y) ln(1-ŷ) would drive predictions toward 1 for both classes; the correct loss should be -[y ln ŷ + (1-y) ln(1-ŷ)] (or equivalently the negative of the expression shown). This is a load-bearing detail because it defines the training objective.","section":"Section 4.2, Eq. (14)"},{"comment":"The text says \"we set R = 5 and P = 3\" in Section 5.2, but later says \"we choose P = 5 and R = 3\" based on Table 4; moreover Table 4 only shows R = 1..4, so the configuration R = 5 is never evaluated in the parameter study. Please clarify which configuration produced Table 3 and reconcile the inconsistency.","section":"Sections 5.2 and 5.3"}],"minor_comments":[{"comment":"The R and P settings are inconsistent across the text and the table; all occurrences should be checked and unified.","section":"Section 5.2/5.3, Table 4"},{"comment":"Equation (15) uses general indices i,j but the surrounding text introduces Sr = {Sr,1,...,Sr,R}; use consistent subscripts to avoid confusion.","section":"Eq. (15)"},{"comment":"The phrase \"As mentioned in Section 3.2\" should refer to Section 3.1.","section":"Section 3.2"},{"comment":"The phrase \"where Max . denotes the maximum operation\" should be typeset as max(·) or similar.","section":"Section 4.2"},{"comment":"The axis title contains a typo: \"Cumulative ocurence proportion\" should be \"Cumulative occurrence proportion\".","section":"Fig. 2"},{"comment":"The model name appears with inconsistent spacing as \"SBN-IV A\" in several places; it should be \"SBN-IVA\".","section":"Table 3 and text"},{"comment":"The average improvement over DN-GAA is reported as 11.2%; verify that (13.8 + 7.99 + 11.8)/3 is correctly derived from the F1 scores in Table 3 and avoid rounding inconsistencies.","section":"Abstract and Section 5.3"},{"comment":"The process of summarizing spoiler keywords per video tag is not described in detail, and no inter-annotator agreement is reported for the human labeling; both would help assess label quality.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the construction of ground truth by keyword filtering; this should be the main focus of the revision. The inconsistency in R/P settings also suggests a careful re-check of the experimental configuration. The paper's contribution is potentially useful for the TSC community, but the current evaluation does not support the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a genuine attempt at a real problem: detecting spoilers in time-sync comments on Chinese video platforms. The model SBN-IVA combines word-level attention, neighbor/keyframe similarity, time decay, and variance-based noise attention into a coherent pipeline. That combination is new for this task, and the dataset is substantial (over 1.5M TSCs across three categories). The paper is transparent about design choices and includes a useful attention visualization. If the reported numbers are reliable, the gain over the DN-GAA baseline is meaningful.\n\nBut there is a load-bearing problem with the evaluation. The ground truth is built by first keyword-filtering TSCs and then having humans label only those candidates. Everything else is automatically treated as non-spoiler. That means genuine spoilers without the preselected keywords are mislabeled as negatives. The model can then achieve high F1 by learning the same shallow keyword cues that generated the labels. This directly affects the central claim of 11.2% average F1 improvement. A model that simply memorizes keywords could artificially beat more principled baselines for exactly this reason.\n\nSecondary issues reinforce this: the keyframe criterion is justified from a cumulative distribution computed on the same keyword-derived labels, there are no error bars or significance tests despite reporting averages over 10 runs, the loss function in Eq. 14 is missing the negative sign, and the hyperparameter settings are inconsistent between Section 5.2 (R=5, P=3) and the paragraph after Table 4 (P=5, R=3). These are fixable, but together they weaken confidence in the experimental pipeline.\n\nThe model logic itself is coherent and the problem is worth solving. I would not cite this as a reliable empirical result until the label bias is addressed, but I would send it to a serious referee because the task is under-explored and the architecture is plausible. A conditional acceptance with a request for data/code release, a properly sampled gold set, and significance testing is the right call.","headline":"A plausible model for an under-explored task, but the keyword-filtered ground truth undermines the headline F1 claim and needs to be fixed before the results can be trusted.","tokens_in":15429,"tokens_out":1805,"would_cite":false,"duration_ms":17396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that spoilers in time-sync comments can be detected by comparing each comment's semantic similarity to its preceding neighbors with its similarity to future highlight 'keyframes,' and reports average F1 gains of 11.2%…","keywords":["Spoiler detection","Time-sync comments","Attention mechanism","Opinion mining","Interactive Variance Attention","Similarity-Based Network","Keyframe similarity","Short-text classification"],"falsifier":"A decisive check: take a random sample of TSCs from each category, have annotators label them in full video context without any keyword pre-filter, and see how many of the newly revealed spoilers contain none of the paper's preselected keywords; a large share would mean the reported F1 advantage is partly an artifact of what the labels could see.","tokens_in":1675,"feed_emoji":"📺","tokens_out":2744,"duration_ms":100091,"temperature":0.7,"pith_summary":"This paper tries to establish that spoilers in time-sync comments, the synchronized scrolling comments on Chinese video sites, can be reliably detected by exploiting their timing and social context rather than only their wording. The proposed model, SBN-IVA, encodes each short comment with a word-level attentive Bi-LSTM, then computes two semantic similarities: how close the comment is to the few comments immediately before it, and how close it is to 'keyframes,' the densest comment periods in the final quarter of the video. The more a comment resembles its neighbors and the less it resembles future keyframe commentary, the less likely it is a spoiler. The authors report F1-scores of 0.850 on TV series, 0.811 on movies, and 0.825 on sports, about 11.2% higher on average than the best baseline. If correct, this gives video platforms a way to hide spoilers while letting non-spoiling commentary through.","feed_headline":"Spoiler detector beats best prior model by 11.2% F1","feed_subtitle":"On TV-series, movies, and sports, F1 reaches 0.850, 0.811, and 0.825 across tested categories.","key_machinery":"The load-bearing object is the neighbor-versus-keyframe similarity gap, sigmoid(G^Ksim_i - G^Nsim_i): a comment is scored as a spoiler when its semantic content resembles what people will say at the video's highlight moments, the keyframes, more than it resembles the ongoing conversation immediately before it. Supporting machinery includes a word-level attentive Bi-LSTM encoder that weights spoiler-relevant words, a time-decay function exp(-$\\beta$(t_Fr - t_i)) over R former neighbors, keyframe selection by taking the P densest 10-second windows in the last quarter of the video, and Interactive Variance Attention, which downweights neighbors whose similarity distribution to the other neighbors is highly concentrated, the signature of noise.","core_discovery":"The paper claims that a TSC's spoiler status can be read off the asymmetry between two similarities: neighbor similarity, a time-decayed weighted average of cosine similarities to the R preceding TSCs, and keyframe similarity, the maximum similarity to P keyframes selected as the densest TSC windows in the final quarter of the video. SBN-IVA predicts the spoiler probability as sigmoid(keyframe similarity minus neighbor similarity), trained with binary cross-entropy. It adds Interactive Variance Attention, which computes for each neighbor the variance of its normalized similarities to all other neighbors and softmax-normalizes the inverse variances, so noisy comments with concentrated similarity distributions receive smaller weights. On datasets crawled from Chinese video platforms across TV-series, movies, and sports, the authors find SBN-IVA beats keyword matching, LDA, an SVM-based method, and a genre-aware deep baseline on precision, recall, and F1. The paper itself flags (Section 6) that the method is not designed for extreme users who flood the screen with spoilers, since the neighbors are then all spoilers themselves.","pith_inferences":["If the neighbor-versus-keyframe asymmetry is the true signal, the same approach should transfer to other languages and platforms that show synchronized comments, as long as highlight windows can be identified from comment density.","The keyword-first labelling procedure may inflate the reported gains, because spoilers that avoid all preselected keywords are never inspected and are silently treated as non-spoilers; a label set built by reading a random sample without keyword prefiltering would test this.","The variance-based noise detector is a general mechanism that could be applied to other temporally ordered short-text streams, such as live chats or sports play-by-play, wherever noise is a known problem.","The extreme-spoiler-flood case the paper leaves open could be addressed by extending the model with user-level features that discount neighbors from accounts that have posted many spoilers, though the paper itself only lists user IDs as future work."],"forward_implications":["The model computes each spoiler score from R preceding comments and P keyframe windows, so the test-time inputs are local rather than requiring the full future comment stream.","Removing the timestamp decay term lowers F1 by 3.31 to 7.24 percent across categories, according to the paper's SBN versus SBN-WT comparison, indicating the real-time property is doing measurable work.","Adding Interactive Variance Attention raises F1 by 3.31 to 9.11 percent over the plain similarity network, according to the paper's SBN versus SBN-IVA comparison, indicating noise suppression is a measurable part of the gain.","Validation results show F1 rises as the number of former neighbors R and keyframes P increases, up to R = 3 and P = 5 in the reported sweep, after which the authors stop for time efficiency.","The same model is reported to be best on precision, recall, and F1 across TV-series, movies, and sports, suggesting the neighbor-versus-keyframe contrast transfers across genres."],"supporting_citations":[{"why":"Supplies the definition and initial properties of time-sync comments, including their short-text nature, which motivates the word-level attentive encoder.","marker":"[29]"},{"why":"Establishes that noise comments have weak semantic relevance to their surroundings, the premise that Interactive Variance Attention relies on to downweight them.","marker":"[34]"},{"why":"Shows that highlight shots of a video coincide with the densest time-sync comment periods, which the model uses to select keyframes in the final quarter.","marker":"[30]"},{"why":"Provides the genre-aware attention deep model that serves as the strongest baseline, against which the 11.2% average F1 gain is measured.","marker":"[3]"},{"why":"Supplies the herding-mentality account of how consecutive time-sync comments influence each other, motivating the neighbor-similarity side of the model.","marker":"[10]"}],"fun_headline_variants":["Spoiler detection sees 11.2% F1 leap with variance attention","Interactive variance attention boosts spoiler detect by 11.2% F1","Time-sync comment spoilers unmasked via similarity asymmetry","SBN-IVA: 11.2% better at spotting spoilers in live comments","Variance-aware AI flags spoilers in time-sync comments"],"cache_read_input_tokens":17536,"weakest_assumption_plain":"The ground-truth labels in Section 5.1 assume that any spoiler contains at least one of the human-preselected spoiler keywords, because TSCs that match no keyword are never inspected by the annotators and are treated as non-spoilers.","fun_headline_variants_meta":{"raw":{"variants":["Spoiler detection sees 11.2% F1 leap with variance attention","Interactive variance attention boosts spoiler detect by 11.2% F1","Time-sync comment spoilers unmasked via similarity asymmetry","SBN-IVA: 11.2% better at spotting spoilers in live comments","Variance-aware AI flags spoilers in time-sync comments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00082,"raw_usage":{"total_tokens":3602,"prompt_tokens":972,"completion_tokens":2630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2532}},"tokens_in":588,"tokens_out":2630,"duration_ms":18391,"temperature":1.0,"reasoning_tokens":2532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:13:01.513952+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check: take a random sample of TSCs from each category, have annotators label them in full video context without any keyword pre-filter, and see how many of the newly revealed spoilers contain none of the paper's preselected keywords; a large share would mean the reported F1 advantage is partly an artifact of what the labels could see.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the definition and initial properties of time-sync comments, including their short-text nature, which motivates the word-level attentive encoder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that noise comments have weak semantic relevance to their surroundings, the premise that Interactive Variance Attention relies on to downweight them."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that highlight shots of a video coincide with the densest time-sync comment periods, which the model uses to select keyframes in the final quarter."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the genre-aware attention deep model that serves as the strongest baseline, against which the 11.2% average F1 gain is measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the herding-mentality account of how consecutive time-sync comments influence each other, motivating the neighbor-similarity side of the model."}],"review_version":1}