{"id":"8b63b4e7-6cc7-471c-9b81-83bea31ca1f2","arxiv_id":"2412.19794","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper builds a 17,435-clip video tampering benchmark and finds that many vision-language models score near trivial-guessing levels, while model size shows no clear link to robustness.","lead":"MVTamperBench applies five simple video edits, dropping, masking, rotation, repetition, and substitution, to 3,487 source videos and asks 45 vision-language models whether each edited clip looks tampered with. It is a new evaluation resource for a practical question: can multimodal models flag manipulated video before it spreads through moderation and social platforms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Several models score exactly 0.667 on every tamper type, which is the F1 of an always-Yes policy on this balanced binary set, so the reported rankings partly measure response bias rather than tamper detection.","rationale":"The reader's weakest-assumption diagnosis is correct and is the most load-bearing issue in the paper: the binary evaluation protocol can reward a constant-Yes response with F1=0.667, and Table 7 contains several models sitting exactly at that value on every tamper type. Since the central claims are a ranking of 45 models and a claim that size does not correlate with robustness, any score that is indistinguishable from a trivial policy undermines those claims unless the authors can show the models are not simply answering 'Yes' throughout. The arithmetic is directly verifiable from the paper's own stated balanced protocol and Table 7, so this is not a matter of differing theoretical taste; it is a mechanical confound. I agree with the reader that this is fixable by reporting baselines and answer distributions, so a conditional verdict is appropriate rather than outright rejection. I would keep the reader's CONDITIONAL verdict unchanged. A secondary inconsistency, that Section 4.2.2 states Figure 12 shows larger models generally do better while Figure 5 is said to show no correlation, is noted but does not change the primary concern.","tokens_in":23572,"tokens_out":2882,"duration_ms":32652,"concrete_test":"Run the released evaluation code on the released balanced dataset with two dummy predictors: a constant-Yes predictor and a constant-No predictor. If constant-Yes yields F1=0.667 on every tamper type, then every Table 7 row at exactly 0.667 is a trivial-policy result. Then recompute all model F1 scores and the size-correlation after removing models whose answer distribution is >99% one-sided, and report whether the Pearson r=0.05 conclusion survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central claims — substantial variability across tamper types and no correlation between parameter count and robustness — depend entirely on the model F1 scores in Table 7. Section 4.1 states that for each tampered video the corresponding non-tampered video is included, giving a balanced binary set. Under that protocol, a model that always answers 'Yes' to the tampering question has TP=N, FP=N, FN=0, so precision=0.5, recall=1.0, and F1=0.667. Table 7 contains multiple models with exactly this value on every row: Chat-UniVi-7B (0.666/0.666), molmo-7B-O-0924 (0.667), Video-LLaVA-7B-HF (0.667), and molmo-72B-0924 (0.667). These models are therefore indistinguishable from a trivial always-Yes policy; they provide no evidence of detecting tampering. Similarly, scores near 0.001-0.01 approach an always-No policy. No trivial baselines are reported anywhere in the paper, and no per-model answer distributions are given, so readers cannot tell whether a listed score reflects detection ability or a fixed response tendency. Section 3.3 even concedes that sub-1-second tampering can be missed by some sampling mechanisms, which could push models toward always-No. Because the paper uses these scores to rank models and to compute the Pearson r=0.05 size correlation, the headline conclusions are not yet supported. The issue is concrete and checkable, and the fix is straightforward: report always-Yes, always-No, and random baselines along with the per-model Yes/No distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MVTamperBench, a benchmark built from 3,487 MVBench videos expanded into 17,435 clips by applying five tampering techniques: dropping, masking, substitution, repetition, and rotation. It evaluates 45 MLLMs on a balanced binary tampering-detection task with a fixed structured prompt and reports macro-averaged F1, claiming substantial variability across tampering types and no correlation between parameter count and robustness. The benchmark, code, and evaluation harness are publicly released.","tokens_in":23791,"tokens_out":4178,"duration_ms":39487,"significance":"If the F1 scores are trustworthy, the benchmark would be a useful resource, and the claim that architecture and training matter more than parameter count for tamper robustness would be an important empirical finding. The paper's strengths include broad model coverage (45 models across 15+ families), a unified evaluation protocol, and public release of code and data. At present, however, the absence of trivial baselines and per-model answer distributions means the central empirical claims are not yet supported. The concern is concrete, checkable, and correctable within the scope of a revision.","major_comments":[{"comment":"Under the balanced binary protocol described in §4.1, an always-Yes policy achieves F1 = 0.667 and an always-No policy achieves F1 = 0. In Table 7, Chat-UniVi-7B, molmo-7B-O-0924, Video-LLaVA-7B-HF, and molmo-72B-0924 sit at exactly 0.667 for every tampering type, while low-scoring models such as Phi-3-Vision (0.001) and llava-onevision-qwen2-0.5b-ov (0.001) are statistically indistinguishable from an always-No policy. Because no always-Yes, always-No, or random baselines are reported, and no per-model Yes/No distributions are given, readers cannot determine whether the listed scores reflect tamper-detection ability, response bias, or frame-sampling artifacts. This directly undermines the variability and size-correlation claims built on the same F1 values.","section":"§4.1, Table 7"},{"comment":"The paper concedes that sub-1-second tampering 'could be overlooked by certain model sampling mechanisms.' Since all five tampering types are applied as a single 1-second midpoint segment, a model that samples sparsely may never see the altered interval, making its 'No' responses a property of the sampler rather than of tamper-detection ability. The manuscript should either report per-model frame coverage or sampling rates, or include a control where the tampered interval is guaranteed to be sampled, before attributing low scores to lack of robustness.","section":"§3.3"},{"comment":"The Pearson r = 0.05 between parameter count and overall F1 is computed on scores that include many values exactly matching trivial policies. This correlation is not informative without baselines, because it conflates response bias with detection ability. In addition, §4.2.2's size-category analysis (Figure 12) states that 'larger models generally achieve higher F1 scores' while the same section claims no significant correlation, and no confidence interval or p-value is reported. Please recompute the correlation after excluding or baseline-correcting trivial-policy models and report an effect size with uncertainty.","section":"§4.2, Figure 5"},{"comment":"The category boundaries are internally inconsistent: the text reports a 0.25 quantile F1 of 0.071 and a 0.75 quantile F1 of 0.846, then says these are rounded to the nearest integers to obtain low-performing models with F1 < 0.01 and high-performing models with F1 > 0.8. The value 0.071 does not round to 0.01 under any standard rounding rule. The low/moderate/high categorization should be re-derived from a clearly stated rule, and Figures 6–8 regenerated accordingly.","section":"§4.2.1"},{"comment":"The design choices (1-second duration, midpoint placement, structured prompt) were selected after ablations on the same 45 models whose rankings are then reported. This creates a mild selection effect: the reported rankings are for the configuration that maximizes performance on the benchmark's own model set. At minimum, the paper should state explicitly that the prompt and duration configuration is part of the benchmark definition, and evaluate sensitivity by reporting rankings under an alternative fixed configuration (e.g., 2-second tampering or the generic prompt) for a subset of models.","section":"Appendix A.2"}],"minor_comments":[{"comment":"The contributions list contains a duplicated fragment: 'across MLLM families. across MLLM families.' Please remove the repetition.","section":"Abstract/§1"},{"comment":"The Limitation section discusses the scope of tampering types and dataset diversity but does not mention the baseline/response-bias issue that affects the validity of the reported scores; this omission should be addressed.","section":"Section 6"},{"comment":"The sentence beginning 'We the growing interest area in MLLMS...' is grammatically incomplete and should be rewritten.","section":"Appendix A.1"},{"comment":"The table uses inconsistent tampering-type names (Drop, Mask, Repeat, Rotate, Substitute) while the main text uses Dropping, Masking, Repetition, Rotation, and Substitution; please align the terminology.","section":"Table 7"},{"comment":"The statement that 'even MLLMs exceeding 70B parameters suffer severe performance drops' is not clearly supported by Table 7, where the 72B models range from low (0.044) to moderate (0.352) to high (0.667) but none is among the top performers; please clarify which models and which drop are meant.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark artifact and the model sweep are valuable, and the issues raised here are fixable in revision. I would not reject the paper, but the central analytical claims should not be accepted without reporting trivial baselines, per-model answer distributions, and a re-analysis of the size correlation. The already-stated acceptance at ACL 2025 does not alter this assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Briefly: the benchmark artifact is real and useful, but the paper's headline findings are not yet supported. The stress-test concern is correct. Several models in Table 7 sit at exactly 0.667 on every tamper type, which is the F1 of an always-Yes policy under the balanced protocol; others hover near 0.00, which is an always-No policy. No trivial baselines are reported, and no per-model answer distributions are given, so those rows tell you about response bias, not tamper detection. That directly undercuts the 'substantial variability' claim and the r=0.05 size correlation.\n\nWhat is genuinely new: a systematic tampering-robustness benchmark built on MVBench, 17,435 tampered clips, five operators, 45 models, code and data released. No prior benchmark in the cited literature evaluates MLLM robustness to video tampering. That is a real contribution, and I will probably cite it for the dataset.\n\nThe paper also contradicts itself: Figure 5 reports Pearson r=0.05 (no correlation), while Figure 12's caption says 'Larger models generally tend to perform better.' The text tries to hold both, but the abstract's no-correlation claim is the headline. And the design choices — 1-second midpoint tampering, the structured prompt — were chosen after ablating on the same 45 models whose rankings are then reported. That is a mild circularity; it would be fine if the rankings were only used to illustrate, but they are the main result.\n\nBottom line: this is a benchmark paper for people building or testing video MLLMs in content moderation and surveillance. The dataset is worth engaging with; the evaluation protocol needs a re-analysis first. The problems are checkable and fixable: report always-Yes, always-No, and random baselines, give per-model Yes/No distributions, and soften the size claim. It deserves a serious referee, and I'd bring it to a reading group as a clean example of why trivial baselines matter.","headline":"A genuinely useful tampering benchmark whose headline findings are not yet supported because several model scores are indistinguishable from an always-Yes policy.","tokens_in":24509,"tokens_out":3806,"would_cite":true,"duration_ms":36305,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark of 45 video AI models finds that model size does not buy tamper resistance.","keywords":["video tampering detection","multimodal large language models","MLLM robustness","temporal coherence","frame manipulation","benchmark evaluation","F1 score","model scaling"],"falsifier":"Rerun the 45 models on the same videos with the 1-second tampered segment removed but the labels kept, or with labels shuffled; if many models keep F1 near 0.667 (the always-Yes score under balanced classes) or if the model ranking flips, the benchmark is measuring response bias and frame sampling rather than tamper detection.","tokens_in":23254,"feed_emoji":"🎥","tokens_out":7457,"duration_ms":65579,"temperature":0.7,"pith_summary":"Multimodal LLMs that watch video can be silently fooled when a single second of footage is rotated, blacked out, swapped, repeated, or deleted, and until now no benchmark measured how often. MVTamperBench takes 3,487 source videos, creates five tampered versions of each (over 17,000 clips across 19 video tasks), and asks 45 vision-language models a simple yes/no question: was this video tampered with? The paper's central finding is that resilience varies sharply across models and tamper types, and that model size is essentially unrelated to detection performance (Pearson $r = 0.05$). If the benchmark holds up, it gives the field a concrete ranking of which architectures resist manipulation and shifts attention from scaling parameter counts to architectural and training choices.","feed_headline":"Bigger isn't safer: 45 video AI models fail a tampering test","feed_subtitle":"Five one-second edits—rotation, masking, substitution, repetition, dropping—reveal that architecture, not size, decides resistance.","key_machinery":"The load-bearing mechanism is the benchmark pipeline itself: each of 3,487 MVBench videos is edited by five tampering operations (dropping, masking, rotation, substitution, repetition), each applied to a fixed 1-second segment at the video's midpoint, yielding 17,435 clips. The evaluation frames detection as a binary multiple-choice question—'Does this video exhibit any signs of tampering...?'—with balanced tampered/non-tampered pairs, scored by per-tamper-type F1 and macro-averaged into an overall F1. The five operations are deliberately chosen to break spatial coherence (rotation, masking) or temporal coherence (dropping, repetition, substitution). The framework is modular and integrates with VLMEvalKit, so new tamper types and datasets can be added without changing the protocol.","core_discovery":"On its own terms, the paper claims that MVTamperBench is a valid robustness test and that, under that test, current MLLMs show substantial variability: a few models (VILA1.5-40B, the InternVL2.5 family, several Molmo variants) stay above 0.8 F1 across all five tamper types, while many models fall below 0.2 and some near 0.001. Dropping and repetition—temporal disruptions—are the hardest manipulations overall, masking the easiest, and task categories requiring temporal reasoning (counterfactual inference, action prediction) degrade most. Model size shows no meaningful correlation with overall F1 ($r = 0.05$), and within families such as Molmo the smaller variant outperforms the larger one, which the paper reads as evidence that architecture and training strategy, not parameter count, determine tamper resistance. The authors also report that structured prompts are essential: generic and chain-of-thought prompts inflate false positives and collapse F1 for many models.","pith_inferences":["If the size-independence result generalizes, tamper robustness should be tracked as its own evaluation axis in model releases, like safety or hallucination scores, rather than assumed to improve with scale.","The 1-second, midpoint-fixed tamper means scores may depend on each model's frame-sampling schedule; a natural extension is to report frame coverage or detection latency to separate sampling artifacts from genuine perception.","The binary yes/no protocol could be extended to localization and classification of the tamper type, which would reward models that actually perceive the manipulation rather than answer from global video statistics.","Because the source videos all come from MVBench, the ranking may not transfer to user-generated or surveillance footage; re-running the benchmark on those corpora is a direct test of whether the vulnerability pattern holds."],"forward_implications":["Model size is not a useful predictor of tamper resistance: a 1B model (MolmoE-1B) beats most 7B–90B models, while several 72B models score below 0.1.","Temporal tampering—dropping and repetition—is the hardest category for most models, while masking is the easiest, indicating current MLLMs lean on spatial rather than temporal cues.","High-performing families (InternVL2.5, VILA1.5) stay consistent across all five tamper types, suggesting tamper resistance is an architectural and training property rather than a per-task fluke.","The benchmark's structured prompt design matters: generic or chain-of-thought prompts sharply lower F1 and raise false positives, so prompt choice alone can change a model's apparent robustness.","Because the code and data are released and modular, the same protocol can be applied to new models, new tamper types (noise, shuffling, deepfakes), and new domains such as surveillance or clickbait detection."],"supporting_citations":[{"why":"Supplies all 3,487 source videos and the 19 task categories that MVTamperBench tampers and evaluates.","marker":"Li et al., 2024e"},{"why":"The VLMEvalKit toolkit the benchmark integrates with to run the 45 models reproducibly.","marker":"Duan et al., 2024"},{"why":"Survey of deepfake and video-tampering detection that motivates the five real-world tampering scenarios.","marker":"Kingra et al., 2023"},{"why":"MMBench-Video, the long-form video benchmark whose lack of adversarial testing this work positions itself against.","marker":"Fang et al., 2024"},{"why":"Video-MME, the video MLLM evaluation benchmark cited as omitting tampering-specific tasks.","marker":"Fu et al., 2024"},{"why":"BLINK, the image-based reasoning benchmark that represents the static-evaluation baseline this work extends to video tampering.","marker":"Fu et al., 2025"}],"fun_headline_variants":["Bigger AI not safer: 45 video models face tampering test","Video AI fails tampering: New benchmark tests 45 MLLMs","MVTamperBench: 45 video models, 5 tampering tricks, huge gaps","Tampering test exposes video AI: Bigger not better, says study","Video models vary wildly under edits—size doesn't predict safety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's central claim assumes that a model's binary 'tampered or not' answer reflects whether it actually saw the one-second manipulated segment, rather than the model's tendency to answer Yes or No, or which frames it happened to sample.","fun_headline_variants_meta":{"raw":{"variants":["Bigger AI not safer: 45 video models face tampering test","Video AI fails tampering: New benchmark tests 45 MLLMs","MVTamperBench: 45 video models, 5 tampering tricks, huge gaps","Tampering test exposes video AI: Bigger not better, says study","Video models vary wildly under edits—size doesn't predict safety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1542,"prompt_tokens":1028,"completion_tokens":514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":415}},"tokens_in":644,"tokens_out":514,"duration_ms":5776,"temperature":1.0,"reasoning_tokens":415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:50:52.032790+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the 45 models on the same videos with the 1-second tampered segment removed but the labels kept, or with labels shuffled; if many models keep F1 near 0.667 (the always-Yes score under balanced classes) or if the model ranking flips, the benchmark is measuring response bias and frame sampling rather than tamper detection.","supporting_citations":[],"review_version":1}