{"id":"87073669-1535-4e41-a94c-60e3a688597d","arxiv_id":"2508.20546","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A four-modality video hate speech detector using cross-modal attention with on-screen text as query achieves state-of-the-art macro F1 of 0.874 on HateMM.","lead":"Researchers built a system that detects hate speech in videos by combining four information channels: spoken words, audio tone, visual frames, and text shown on screen. On a public benchmark it achieves the best reported balanced F1 score, and finds that letting the on-screen text query the other channels works best.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The query-key selection evidence is internally inconsistent: Tables 6/7 report M-F1 0.877/0.884 for the selected configuration while Table 3 lists 0.837/0.846 for the same CMA-LF/CMA-S setups, so the best-configuration claim and the 0.874 headline rest on an unclear model identity and single-seed…","rationale":"In good faith, the paper contributes a genuinely multimodal HSD system, releases code, and includes extensive ablation material; the central claim is empirical, not formal. The reader's weakest assumption is that the O-query/TAV-key configuration, selected on HateMM validation data with a single seed, generalizes and that selection bias does not explain the 0.874 M-F1. That concern is real, but I find a more concrete and more load-bearing problem: the tables that are supposed to justify the configuration choice do not align with the architecture results in Table 3. The same configuration is reported as 0.846/0.837 in Table 3 but as 0.884/0.877 in Tables 7/6. Because Table 3's standard deviations are tiny, this is not a seed artifact; it indicates either a mislabeling of which architecture produced Tables 6/7 or a substantial unreported difference in the experimental setup. If Tables 6/7 are actually MM-HSD runs, then the paper performed a 28-configuration single-seed search on the final model and selected the best validation performer; in that case the reported test number is the result of a selection procedure that should be validated with multiple seeds and a proper selection rule. If Tables 6/7 are CMA-S/CMA-LF runs, then the reported numbers contradict Table 3 and the transfer of the best query-key configuration to MM-HSD is not established. Either way, the central claim that O-as-query/TAV-as-key is the best configuration and that MM-HSD achieves 0.874 M-F1 rests on evidence that is internally inconsistent. This does not warrant rejection, because the discrepancy is addressable by re-running the configuration search with multiple seeds and by clearly labeling the architecture for each table. It does warrant a conditional verdict, which is what the reader already recommended. I therefore leave the verdict unchanged but flag that the condition is more specific than generic selection bias: the reproducibility of the query-key selection evidence itself is at stake.","tokens_in":19739,"tokens_out":12642,"duration_ms":117847,"concrete_test":"Re-run the 28 query-key configurations from Tables 6/7 under each of the three architectures (CMA-S, CMA-LF, and MM-HSD) with 5 seeds on the same folds. Check (a) whether the O/TAV entries reproduce Table 3 (0.846/0.837) or Tables 6/7 (0.884/0.877); (b) whether TAV-O remains the best MM-HSD configuration by mean validation M-F1 with a margin above 0.01 over the runner-up; and (c) whether the held-out test M-F1 of the configuration selected this way stays above the HCC1 baseline of 0.848. If TAV-O is not selected or the margin shrinks below 0.01, the headline claim should be reported as configuration-dependent rather than a general superiority result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.5 says the query-key search was run on CMA-S and CMA-LF with a single seed and that the full results are in Tables 6 and 7. Table 3, however, reports CMA-S† with M-F1 0.846 (.006) and CMA-LF† with M-F1 0.837 (.024) for the selected O-query/TAV-key configuration. Table 6 (described as late fusion) and Table 7 (described as raw-input fusion) list the same configuration as 0.877 and 0.884, respectively. These numbers are far closer to the MM-HSD value of 0.874 than to the CMA-S/LF values; a 0.038 gap with a reported std of 0.006 cannot be a seed effect. The reader therefore cannot determine which architecture produced Tables 6/7. If Tables 6/7 are actually MM-HSD, then the final configuration was chosen from 28 single-seed MM-HSD runs on the same validation folds, and no multi-seed comparison of alternative configurations exists. If they are CMA-S/CMA-LF, the values contradict Table 3 and the transfer of the O-query/TAV-key choice to MM-HSD is unsupported. Either way, the central design claim (on-screen text as query is best) and the headline 0.874 M-F1 depend on a selection procedure whose model identity and variance are not consistently reported. This is not a question of disagreeing with the consensus; it is a concrete reproducibility gap in the paper's central comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents MM-HSD, a multi-modal hate speech detection model for videos that integrates four modalities: speech transcript (T), audio (A), video frames (V), and on-screen text (O). The proposed architecture applies cross-modal attention (CMA) to the raw modality embeddings and concatenates the CMA output with separately encoded per-modality representations before classification. The authors report that the best configuration uses on-screen text as the attention query and the concatenation of transcript, audio, and video as the key, and they report a macro-F1 of 0.874 on the HateMM benchmark, outperforming published baselines. The paper also contributes an ablation study of modality subsets, query-key configurations, early versus late fusion, and computational efficiency, and it releases the code.","tokens_in":20132,"tokens_out":9032,"duration_ms":82728,"significance":"If the reported benchmark result holds, this is a useful and timely empirical contribution to video-based hate speech detection, a relatively understudied task. The inclusion of on-screen text as a separate modality, the systematic query-key analysis, the released code, and the explicit comparison against several recent baselines are strengths that would make the paper valuable to the community. The reported gap to prior state-of-the-art macro-F1 (0.874 versus 0.848) is practically meaningful. However, the central empirical claim is currently clouded by internal inconsistencies in the reported results and in the experimental protocol, so the significance of the contribution cannot be fully assessed until these issues are resolved.","major_comments":[{"comment":"The results in Tables 6 and 7 are inconsistent with Table 3 for the same selected configuration. For query O and key TAV, Table 6 reports M-F1 = 0.877 and Table 7 reports M-F1 = 0.884, while Table 3 lists CMA-LF† at 0.837 (std 0.024) and CMA-S† at 0.846 (std 0.006) for the same configuration. Since the captions identify Table 6 as late-fusion CMA and Table 7 as raw-input CMA, the reader expects these values to agree with the corresponding rows of Table 3; a gap of 0.038–0.047 is far larger than the reported standard deviations and cannot be attributed to seed variation. The reader therefore cannot determine which architecture actually produced Tables 6 and 7, and the headline 0.874 M-F1 and the design claim that O-as-query is best rest on an unidentified model. Please clarify the exact model behind each table and provide multi-seed mean and standard deviation for the configuration search.","section":"Section 5.5 and Appendix A, Tables 6 and 7, versus Table 3"},{"comment":"The reported data-split counts do not add up. Section 5.1 states that 5-fold cross-validation is performed on 85% of the data, with each fold containing 698 training and 175 validation samples, while Section 5.6 states that inference is performed over 155 test samples. For the 1083 labeled videos in HateMM, these numbers imply different totals: 873 videos for train plus validation, 920.55 for 85% of the data, and 1028 for 698+175+155. Please report the exact number of videos used after preprocessing, describe how the split was constructed, and state how many videos were discarded and why. Without this information, the comparison in Table 2 cannot be reproduced.","section":"Section 5.1 and Section 5.6"},{"comment":"The query-key configuration search is conducted with a single seed and on validation folds, and the selected configuration is then adopted for MM-HSD. Section 5.5 states that the experiments in Tables 6 and 7 were performed in a single-seed setup. If those tables are actually MM-HSD runs, the final configuration is selected from 28 single-seed validation results on the same benchmark, with no multi-seed comparison of alternative configurations. If they are CMA-LF and CMA-S runs, the transfer of the O-query/TAV-key choice to MM-HSD is not directly supported. Either way, the claim that on-screen text is the best query and the headline 0.874 M-F1 need multi-seed mean and standard deviation for at least the top configurations, and the selection procedure should be described in a way that makes clear whether test-set information was used.","section":"Section 5.5 and Section 4.1"},{"comment":"The state-of-the-art comparison in Table 2 is based on scores taken from the original papers rather than re-running baselines under the identical protocol. Because the paper uses a specific 85/15 split with 5-fold cross-validation, differences in data partitioning, preprocessing, and evaluation code can materially affect macro-F1. Please either re-run the prior baselines under the same split and report mean and standard deviation, or explicitly state the splits used by each baseline and justify why the comparison is fair. At minimum, a significance test should be reported for the difference between MM-HSD (0.874, std 0.009) and the previous best HCC1 result (0.848), and the comparison with the single-run result of TCE-DBF [71] should be qualified accordingly.","section":"Table 2 and Section 5.1"}],"minor_comments":[{"comment":"The appendix describes Table 7 as 'early fusion (model II)', but Figure 2 labels model II as 'Late Fusion with CMA as Additional Modality' and model III as 'Early Fusion with CMA as Unique Feature Extractor'. Please make the model labels consistent between the figure and the appendix.","section":"Appendix A, paragraph before Table 6"},{"comment":"The sentence 'M-F1 score of 0.878 (MM-HSD) against 0.846 (w/o CMA)' appears to report accuracy values, since Table 3 lists MM-HSD M-F1 = 0.874 and w/o CMA M-F1 = 0.842. Please correct the metric labels.","section":"Section 5.4, sentence citing Table 3"},{"comment":"The value '0.82813' in the P(H) column appears to be a typo; it should likely be a three-decimal value consistent with the other entries.","section":"Table 7, TVA row"},{"comment":"The reported 'very strong positive correlation of 91%' between CMA-S and CMA-LF performance gains does not state the type of correlation coefficient or its statistical significance. Please add this detail.","section":"Section 5.5, correlation statement"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a multimedia or social-media analysis venue, and the empirical contribution is potentially useful. However, the internal inconsistencies in the appendix tables, the data-split counts, and the single-seed configuration search are load-bearing for the central claims. I would not reject the paper outright because these issues appear fixable by re-running or re-reporting the experiments, but the authors need to resolve them before the benchmark claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a decent, well-engineered paper with one genuinely new empirical claim—using on-screen text as the query in cross-modal attention beats other query/key configurations for video hate speech detection—and the best published M-F1 on HateMM if the number holds. The paper also ships code, runs 5-fold cross-validation, and does real ablations, including a useful stopword check and a computational cost table. That is more than many submissions in this area do. I'd send it to reviewers, but the central configuration-selection story has a hole that needs patching before the headline result is citable.\n\nWhat's new: the systematic comparison of query/key choices in CMA for video HSD, and the treatment of OCR text as a standalone modality rather than merging it into the transcript. Previous CMA work used text as the only query and ignored on-screen text. The authors show that O-as-query/TAV-as-key is best, and that removing any modality hurts. Those are useful, concrete findings.\n\nThe soft spot is the reproducibility of the selection. Section 5.5 says the query-key search was done on CMA-S and CMA-LF with a single seed, with full results in Tables 6 and 7. But Table 3 reports the same selected configuration for CMA-S and CMA-LF as 0.846 and 0.837 M-F1, while Tables 6 and 7 list 0.877 and 0.884 for what is supposed to be the same architecture. A 0.03–0.04 gap with reported standard deviations of 0.006–0.024 cannot be brushed off as seed noise. The appendix labels also conflict: Section 4.1 points to Table 7 for the MM-HSD design, but Table 7 is captioned as raw-input fusion (CMA-S), and Section 5.5 says the search was on CMA-S/CMA-LF, not MM-HSD. So the reader cannot tell which model produced the configuration that is then transferred to MM-HSD, or whether the final configuration was selected from single-seed runs on the same validation folds. That is a concrete reporting gap, not a nitpick. It does not by itself invalidate the architecture, but it does mean the headline number is currently optimistic in an unquantified way.\n\nOther weaknesses: baseline methods are not re-run under the same protocol, and there is no significance test against prior work. The authors quote TCE-DBF's single-run numbers as-is; that is fine, but it limits the strength of the comparison. To their credit, they are explicit that the model is trained only on HateMM and that generalization to other datasets is future work.\n\nWho this is for: people working on video HSD or multimodal fusion will want to read it; the ablations are informative even if the headline is provisional. I would send it to peer review, but with a request to clarify Tables 6 and 7, add multi-seed validation of the chosen configuration, and ideally re-run the baselines under the same conditions. After those fixes, I would cite it.","headline":"Solid engineering paper with a plausible new finding—on-screen text as query in cross-modal attention—but the configuration-selection evidence is internally inconsistent, so the headline M-F1 needs a reproducibility fix before I'd trust it.","tokens_in":20647,"tokens_out":4476,"would_cite":false,"duration_ms":42631,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"0.874 macro-F1 for video hate detection with on-screen text as query","keywords":["hate speech detection","multimodal fusion","cross-modal attention","video classification","on-screen text","OCR","audio-visual analysis","HateMM"],"falsifier":"Re-run MM-HSD on HateMM while choosing the attention arrangement from the average validation macro-F1 over several random seeds instead of one seed; if the chosen arrangement changes or the test macro-F1 falls below the 0.848 of the previous best method, the claim that on-screen text is the best query is not supported. A second check is to fix the arrangement and evaluate on a different video hate dataset, such as MultiHateClip, where failing to beat a transcript-only model would undercut the generality of the four-modality design.","tokens_in":19603,"feed_emoji":"🎬","tokens_out":16624,"duration_ms":135225,"temperature":0.7,"pith_summary":"MM-HSD is a four-modality model for detecting hate speech in videos, combining the speech transcript, audio, video frames, and on-screen text. The paper's central claim is that using cross-modal attention with on-screen text as the query and the concatenated transcript, audio, and video embeddings as the key gives the best fusion configuration, reaching a macro-F1 of 0.874 on the HateMM benchmark and outperforming the previous best published score of 0.848. This matters because hate in videos often appears in a non-obvious modality: letting a sparse, potentially hate-carrying channel like on-screen text pull context from richer modalities captures dependencies that simple concatenation or late fusion can miss. The authors also present the first systematic query/key comparison for cross-modal attention in video hate speech detection, and ablate every component to show that each modality and the attention block contribute.","feed_headline":"0.874 macro-F1 for video hate detection with on-screen text as query","feed_subtitle":"A four-modal attention model tops HateMM by letting OCR text pull cues from video, audio, and transcript.","key_machinery":"The load-bearing mechanism is Cross-Modal Attention (CMA), an attention operation of the form $\\mathrm{softmax}(Q K^\\top / \\sqrt{d_k})V$ that lets one modality query a sequence of features from other modalities. In MM-HSD the query is the on-screen text embedding and the key/value is the concatenation, along the sequence dimension, of the transcript, audio, and video embeddings; the CMA output is then concatenated with separately encoded unimodal outputs before the final classification head. The design decouples visual context from on-screen text by using a vision transformer for frames and a separate OCR module for the text in the frame, so the query channel is not redundant with the video channel. The role of the CMA block is to let a sparse, sometimes absent, on-screen-text signal absorb contextual cues from the other three modalities and to do so at the embedding level rather than after classification.","core_discovery":"The paper reports that on the HateMM dataset, a tetra-modal pipeline — speech-transcript text embeddings, audio embeddings from a self-supervised speech encoder, video-frame embeddings from a vision transformer, and on-screen text extracted by OCR — fused by a cross-modal attention (CMA) block plus unimodal encoders reaches an unbiased accuracy (mean of class-wise recalls) of 0.878, macro-F1 of 0.874, and hate-class F1 of 0.853 over five runs. The configuration that delivers this is CMA applied to raw embeddings with on-screen text as query and the concatenation of transcript, audio, and video as key, with the CMA output concatenated to the four unimodal encoder outputs before classification. Across the full grid of query/key combinations, this O-query/TAV-key choice is the best, while using on-screen text as part of the key generally hurts performance. Removing any single modality from the full model drops macro-F1 to between 0.815 and 0.845, and replacing CMA by plain concatenation drops it to 0.842, so the paper's claim is that every component contributes and the attention asymmetry is load-bearing.","pith_inferences":["If the query-asymmetry result transfers, it suggests a general heuristic for multimodal fusion: let the modality most likely to carry the target signal be the attention query and let richer context modalities be the keys, rather than always defaulting to text as the query.","Because the configuration search over query-key pairs was run on validation macro-F1 with single-seed runs, the true advantage of the O-query/TAV-key choice over its nearest alternatives may be smaller; averaging configuration selection over seeds or folds is a direct robustness check the paper leaves open.","The same architecture is naturally extendable to frame-level hate localisation by making the attention temporally aware, which would also expose which frames drive a video-level hate verdict.","A testable extension is to swap the transcript channel for speech-to-text output in another language (or a dataset without on-screen text) to see whether the OCR-query design remains optimal, or whether the query should become the modality that is most sparse or most hate-bearing in that new setting."],"forward_implications":["Video hate-speech systems should treat on-screen text as a distinct modality: OCR captures hate conveyed in frames that transcript-only or image-only models cannot see.","Early cross-modal attention plus late fusion of the unimodal encoders beats either alone: the full model reaches 0.874 macro-F1 against 0.846 for CMA used standalone and 0.842 for concatenation without CMA.","The attention asymmetry is a design choice that matters: making on-screen text the query and transcript, audio, and video the key is the best arrangement among all tested query-key pairs.","Every modality pulls its weight in the tetra-modal setting; removing any one of transcript, audio, video, or on-screen text lowers macro-F1, with transcript removal causing the largest drop.","Deployment is plausible once features are precomputed: the final classifier has 4.6 million parameters even though the upstream feature extractors are much larger."],"supporting_citations":[{"why":"Supplies the HateMM dataset used for training, validation, and testing, as well as the earlier unimodal baselines this work builds on.","marker":"[12]"},{"why":"Provides the previous best method (macro-F1 0.848) that MM-HSD compares against and outperforms.","marker":"[40]"},{"why":"Is the only prior model that also incorporates on-screen text and gives a second macro-F1 comparison point.","marker":"[71]"},{"why":"Introduces the cross-modal attention mechanism that MM-HSD adapts to early fusion.","marker":"[11]"},{"why":"Is the prior CMA-based video toxicity detector whose bimodal query design MM-HSD generalizes to four modalities.","marker":"[47]"},{"why":"Provides the hate-speech-tuned text encoder used to embed transcripts and on-screen text.","marker":"[28]"},{"why":"Supplies the OCR engine that extracts the on-screen text modality.","marker":"[52]"},{"why":"Provides the speech-to-text transcription that creates the transcript modality.","marker":"[57]"},{"why":"Supplies the self-supervised audio representation used for the audio channel.","marker":"[5]"}],"fun_headline_variants":["On-screen text as query lifts video hate detection to 0.874 macro-F1","MM-HSD: cross-modal attention with OCR text as query top HateMM at 0.874","Four-modal video hate detection: attention query from on-screen text wins","0.874 macro-F1: let OCR text guide attention in multi-modal hate detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The key premise is that the arrangement chosen by trying many alternatives on the validation portion with single-seed runs — letting on-screen text pull information from transcript, audio, and video — is also the best arrangement on the held-out test set; if that search got lucky on validation, the reported 0.874 macro-F1 could be too high.","fun_headline_variants_meta":{"raw":{"variants":["On-screen text as query lifts video hate detection to 0.874 macro-F1","MM-HSD: cross-modal attention with OCR text as query top HateMM at 0.874","Four-modal video hate detection: attention query from on-screen text wins","0.874 macro-F1: let OCR text guide attention in multi-modal hate detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0005,"raw_usage":{"total_tokens":2487,"prompt_tokens":1029,"completion_tokens":1458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":1368}},"tokens_in":645,"tokens_out":1458,"duration_ms":10906,"temperature":1.0,"reasoning_tokens":1368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:25.667852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run MM-HSD on HateMM while choosing the attention arrangement from the average validation macro-F1 over several random seeds instead of one seed; if the chosen arrangement changes or the test macro-F1 falls below the 0.848 of the previous best method, the claim that on-screen text is the best query is not supported. A second check is to fix the arrangement and evaluate on a different video hate dataset, such as MultiHateClip, where failing to beat a transcript-only model would undercut the generality of the four-modality design.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-supervised audio representation used for the audio channel."},{"cited_title":"Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever","cited_arxiv_id":null,"evidence_quote":"Provides the speech-to-text transcription that creates the transcript modality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the only prior model that also incorporates on-screen text and gives a second macro-F1 comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the cross-modal attention mechanism that MM-HSD adapts to early fusion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the prior CMA-based video toxicity detector whose bimodal query design MM-HSD generalizes to four modalities."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OCR engine that extracts the on-screen text modality."}],"review_version":2}