{"id":"18ef9cd1-3400-4613-b3e2-9836dcdbec8c","arxiv_id":"2501.12681","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Training CLIP-based action recognition models with masked backgrounds or objects lowers their reliance on static scene cues and improves person-focused accuracy on several video datasets.","lead":"This paper tests whether hiding backgrounds and objects during training helps zero-shot action recognition models focus on human actions instead of scene bias. It finds that CLIP-based models do rely on static cues, and that selective masking can shift attention to the person, with mixed but promising accuracy effects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"P-top1 gains may reflect train/eval distribution match for the masking augmentation, not reduced static bias; the central claim needs a control test with held-out masking styles.","rationale":"The paper's central claim is that masking backgrounds or objects during training reduces static bias and focuses CLIP-based zero-shot models on human actions. The primary evidence for this claim is the P-top1 metric, which masks the background at evaluation time. Because the same masking operation is used as a training augmentation, P-top1 measures, at least in part, how well the model handles inputs that match its training distribution. The unmasked baseline has never seen color-filled backgrounds, so its low P-top1 may reflect domain shift rather than static bias. This is a more fundamental threat than the reader's concern about segmentation quality: even with perfect person/object/background masks, the reported P-top1 improvement would not establish the conclusion without controlling for the augmentation-alignment confound. The B-top1 drop, while suggestive, is also insufficient on its own because it involves yet another input corruption (person removal) and could be driven by a general change in feature statistics. The proposed concrete test, evaluating with held-out masking styles, would separate 'learned to use person cues' from 'learned to tolerate the specific coloring augmentation.' If the P-top1 gains do not transfer to unseen masking styles, the central claim would need to be substantially weakened. I therefore keep the reader's CONDITIONAL verdict, but with a different and more load-bearing condition: the authors must rule out the augmentation-alignment confound before the static-bias-reduction claim can be accepted. Secondary issues such as missing error bars, lack of code/data, and tuning masking ratios after seeing results remain valid but are not the decisive concern.","tokens_in":11071,"tokens_out":4729,"duration_ms":50947,"concrete_test":"Train ViFi-CLIP with the background-masking ratio 0.33:0.67 as in Table 1. Evaluate P-top1 under three held-out background corruptions not seen in training: (a) a fixed gray background, (b) Gaussian noise background, (c) backgrounds copied from other videos (Action-Swap style). If the P-top1 improvement over the no-masking baseline persists across all three, the model is genuinely using person cues. If it largely disappears (e.g., the margin shrinks from roughly 14 points to less than 3 points), the reported gain is an artifact of matching the training augmentation. Also report top1 on original unmasked validation videos for both models as a sanity check.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.2 defines P-top1 by masking the background at evaluation (Eq. 2), while the training procedure (Section 3.3) augments videos with the same background-masking operation at the chosen ratio. Consequently, a model trained with masking sees color-filled backgrounds at training time, so its higher P-top1 relative to the unmasked baseline may measure robustness to this specific augmentation, not a reduced tendency to rely on background appearance. The unmasked baseline encounters an out-of-distribution input during P-top1 evaluation, while the masked model encounters an in-distribution one. The B-top1 drop is also not decisive, because it evaluates a different input corruption (person-bbox removal) and could reflect a global shift in feature statistics rather than a targeted reduction of static bias. The paper does not provide a control that varies the masking style at evaluation (e.g., fixed gray, noise, or background swap), nor does it show on original, unmasked videos that decisions are more person-centric via attribution. Thus, even with perfect segmentation, the central claim would not follow: the reported P-top1 gains are confounded by training/eval augmentation alignment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies static bias in zero-shot action recognition for CLIP-based video models. It proposes data-augmentation masking of background regions, object regions, or both during training, and introduces two evaluation metrics, P-top1 and B-top1, computed by masking the background or the person bounding box on validation videos. Experiments with ViFi-CLIP and ActionCLIP on Kinetics400, Mimetics, and SSv2 show that background masking raises P-top1 and lowers B-top1, that masking helps on Mimetics where background bias is weak, and that masking background and objects in different colors can improve SSv2 performance. The paper concludes that masking backgrounds or objects during training reduces reliance on static bias and shifts predictions toward the person.","tokens_in":11260,"tokens_out":4502,"duration_ms":43170,"significance":"If the central claim holds, the paper would provide useful, previously missing evidence about static bias in zero-shot action recognition with vision-language models, and would suggest a cheap, practical augmentation-based mitigation. The paper has concrete strengths: it tests two model architectures and three datasets spanning different bias regimes, introduces interpretable P-top1/B-top1 metrics, and includes a comparison with S3Aug. However, the current experiments do not fully separate the effect of reduced static bias from the effect of training/evaluation augmentation alignment, and one abstract-level claim is contradicted by the paper's own table. With additional controls and more careful statistical reporting, the contribution could become solid; as it stands, the evidence is suggestive rather than conclusive.","major_comments":[{"comment":"The P-top1 evaluation uses the same background-masking operation that is used as a training augmentation, so the reported P-top1 gains over the unmasked baseline are confounded by train/eval distribution matching rather than by a measured reduction in static bias. A model trained with masked backgrounds sees color-filled backgrounds at evaluation and is therefore evaluated in-distribution, whereas the unmasked baseline sees an unusual corruption. To support the central claim, the authors should add a control that varies the masking style at evaluation (e.g., fixed gray, noise, random patches, or background swapping) and show that P-top1 improvements persist across unseen masking styles, or provide attribution evidence on original unmasked videos showing decisions become more person-centric.","section":"§4.2, Eq. (2); §3.3"},{"comment":"The abstract states that masking both background and objects in different colors improves performance for SSv2, but Table 4 shows this is not true for ViFi-CLIP on unseen categories: the novel (unseen) top1 drops from 9.62 at the 1.0:0.0:0.0:0.0 baseline to 9.13 at the 0.5:0.0:0.0:0.5 ratio, a decrease of 0.49. Only ActionCLIP shows a clear unseen-category gain (5.80 to 8.06). The conclusion and abstract should be qualified to state that the benefit is model- and split-dependent, or the claim should be restricted to the seen categories where the evidence is more consistent.","section":"Table 4 and Abstract"},{"comment":"All tables report single runs without error bars or significance tests, and the best masking ratio is selected per model and per dataset after inspecting the results. This makes it difficult to distinguish genuine effects from noise and risks post-hoc selection bias, especially on Mimetics where the validation set has only 713 videos and differences of 1–3 top1 points are reported. The authors should report means and standard deviations over at least three seeds (or over the three base-to-novel splits), and either pre-register the masking ratios or choose them on a separate validation split before reporting final test numbers.","section":"Tables 1–4, §4.1"},{"comment":"The entire method and both evaluation metrics depend on GroundingDINO and SAM masks correctly separating persons, objects, and backgrounds frame by frame, yet the paper provides no quantitative or qualitative validation of segmentation quality. If object detectors miss objects or segment hands incorrectly, P-top1 and B-top1 could measure masking artifacts rather than static bias. The authors should report segmentation accuracy or at least show representative success and failure cases across all three datasets, and discuss how mask errors affect the main conclusions.","section":"§4.1, Masking; Eqs. (2)–(5)"}],"minor_comments":[{"comment":"In Eq. (5), the complement/overline notation appears to be missing: as printed, the two color terms are multiplied by the same mask, so it is impossible to tell whether the background and objects actually receive different colors. Please rewrite the equation with explicit masks for background and object regions.","section":"Eq. (5)"},{"comment":"The S3Aug reference appears twice as [28] and [29]; please merge the duplicate and renumber subsequent references.","section":"References"},{"comment":"There is a typo in 'backgrond bias' in the first paragraph of Section 4.3.","section":"§4.3"},{"comment":"The paragraphs following Table 4 (e.g., 'masking the object bounding box and the object shape', 'a combination of masking') read like bullet fragments; they should be integrated into coherent prose.","section":"§4.4"},{"comment":"The phrase 'P-top1 was significantly lower than B-top1' uses 'significantly' in a colloquial sense; since no statistical test is reported, please rephrase or add the appropriate test.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study, but the main claim about reducing static bias is currently underdetermined by the experimental design because of the training/eval augmentation confound. The SSv2 overclaim in the abstract is a clear accuracy issue that must be fixed. I would like the editor to encourage the authors to add the control experiments and uncertainty estimates outlined in the major comments; with those additions the paper could be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is better than the stress-test note makes it look. The P-top1 metric has a real train/eval confound, but the paper's central claim about reducing static bias survives on the strength of the Mimetics cross-dataset results, which use plain top-1 on original videos.\n\nWhat's new: previous static-bias analyses (Action-Swap, S3Aug, DARK) were all fully supervised. This is the first to ask whether CLIP-based zero-shot action recognition models suffer the same problem and whether masking backgrounds/objects helps. The P-top1/B-top1 pair is a cheap diagnostic that could be useful beyond this paper. The empirical pattern is consistent: across ViFi-CLIP and ActionCLIP, on Kinetics, Mimetics, and SSv2, masking training raises P-top1 and lowers B-top1 relative to the no-masking baseline. The strongest evidence is the cross-dataset zero-shot setting: models trained on 350 Kinetics classes, evaluated on Mimetics (weak background bias) with no masking at test. Masking improves top1 there by 1.6-3.5 points. That result is not confounded by train/eval augmentation matching.\n\nSoft spots, in order of importance. First, the P-top1 confound: P-top1 is computed by masking the background at eval, which is the same operation used in training. The no-mask baseline faces an out-of-distribution input. So P-top1 gains partly reflect distribution matching. The B-top1 drop is less confounded but it's a different corruption, and without a control that varies masking style at eval (gray, noise, background swap) it's hard to separate bias reduction from feature-statistics shift. Second, no error bars or significance tests; several comparisons are within a point or two. Third, mask ratios are selected per model and dataset after looking at results. That's overfitting to the validation set, though the consistency across settings mitigates it. Fourth, the abstract overstates SSv2: ViFi-CLIP's unseen accuracy actually drops slightly with the best SSv2 masking. Fifth, Eq. (5) is miswritten as given (both color terms multiply the same empty intersection). The intent is clear from context, but it's a typo. No code or data are released, which is a reproducibility gap.\n\nWho should read it: anyone working on zero-shot video understanding or static-bias diagnostics. It's not a breakthrough, but it's a solid subfield contribution that opens a question other groups will build on. For peer review, it deserves a serious referee. I would not desk-reject. With error bars, a held-out masking control, and a fix to the abstract and Eq. (5), it would be acceptable.","headline":"Solid extension of masking to zero-shot CLIP action recognition; the P-top1 metric is confounded, but the Mimetics results carry the paper.","tokens_in":11803,"tokens_out":4313,"would_cite":true,"duration_ms":40678,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP-based zero-shot action recognition models lean on background and object appearance, and color-masked training can push them back to the actor.","keywords":["zero-shot action recognition","static bias","background bias","object bias","CLIP","masking augmentation","P-top1","B-top1"],"falsifier":"A concrete falsifier: compute P-top1 and B-top1 after replacing masked regions with randomly sampled patches from other videos instead of uniform colors. If P-top1 gains vanish and B-top1 no longer drops, the reported effect is an artifact of color masking rather than a genuine reduction in static bias.","tokens_in":10855,"feed_emoji":"🎬","tokens_out":5373,"duration_ms":47376,"temperature":0.7,"pith_summary":"This paper asks whether CLIP-based zero-shot action recognition models are secretly recognizing scenes rather than actions. It answers yes: masking the background during training lowers performance on Kinetics400, showing the model had been leaning on background cues, while the same masking improves accuracy on Mimetics, a dataset with weak background bias. On Something-Something v2, where object appearance is the dominant shortcut, masking both background and objects in different colors raises top-1 accuracy. The paper concludes that training with masked static regions pushes models to allocate more attention to the person and the action itself.","feed_headline":"Color-masked training cuts static bias in zero-shot video models.","feed_subtitle":"It lifts Mimetics and SSv2 accuracy by making models look at the actor, not the scenery.","key_machinery":"The machinery is a set of masking operations defined by Eq. (1)-(5): each frame is blended with a randomly sampled color $c$ over the regions to be suppressed, using binary masks produced by GroundingDINO and SAM. Background masking keeps only the person; object masking removes object texture, bounding boxes, or both object and background while preserving object shape. The evaluation uses two new metrics: P-top1 (accuracy when background is masked, so only the person is visible) and B-top1 (accuracy when the person is masked, so only the background is visible). These metrics separate where the model is looking, letting the authors attribute accuracy changes to background or object bias.","core_discovery":"The central discovery is that static bias exists in zero-shot CLIP-based action recognizers and can be counteracted with simple color-masking augmentation. Using GroundingDINO and SAM to segment persons, objects, and backgrounds, the authors mask each region with randomly sampled colors during training. This consistently raises P-top1, a metric that measures accuracy when only the person is visible, while lowering B-top1, the metric for background-only predictions. On Mimetics, background masking improves overall top-1 accuracy by up to 3.47% for ActionCLIP in the cross-dataset zero-shot setting. On SSv2, masking background and object with different colors improves seen-category top-1 from 12.89% to 13.93% for ViFi-CLIP. The authors interpret this as evidence that masking prevents over-reliance on static bias and focuses the model on human action.","pith_inferences":["Beyond the paper, the same masking recipe could serve as a cheap diagnostic for any video-language model: a large gap between P-top1 and B-top1 quantifies static-bias strength without needing background-swap datasets.","A natural next experiment, not run here, is to vary the masking color distribution; the paper samples colors from a standard normal, and one could test whether adversarial or scene-matched colors change the amount of residual bias.","The results suggest a regularization objective: penalize agreement between masked-background and masked-person predictions during fine-tuning, which the paper does not propose but its metrics make straightforward."],"forward_implications":["Masking background during training can be adopted as a cheap data-augmentation drop-in for CLIP-based zero-shot action recognition, with no new architecture required.","On datasets with weak background bias, such as Mimetics, background masking improves overall top-1 accuracy, not just person-focused accuracy.","For object-biased benchmarks like SSv2, masking both background and object in different colors is the most effective recipe, preserving object shape while removing texture.","P-top1 and B-top1 provide a low-cost diagnostic for static bias, simpler than background-swapping evaluation protocols.","The optimal masking ratio and masking type differ between models and between seen versus unseen categories, so the augmentation should be tuned per architecture."],"supporting_citations":[{"why":"GroundingDINO supplies the open-vocabulary object detection that locates persons and objects before masking.","marker":"[18]"},{"why":"SAM supplies the segmentation masks for person, object, and background regions used by every masking equation.","marker":"[13]"},{"why":"ViFi-CLIP is one of the two base zero-shot action recognition models fine-tuned in the experiments.","marker":"[24]"},{"why":"ActionCLIP is the other base zero-shot model fine-tuned, providing a temporal-aggregation comparison.","marker":"[31]"},{"why":"Mimetics is the weak-background-bias evaluation set central to showing that masking improves accuracy.","marker":"[32]"},{"why":"Something-Something v2 is the strong-object-bias benchmark used to evaluate object masking.","marker":"[7]"},{"why":"Chung et al. established static bias in fully-supervised action recognition and motivated the masking approach.","marker":"[4]"},{"why":"Kinetics400 is the background-biased dataset used for training and evaluation throughout the study.","marker":"[11]"},{"why":"The infoNCE loss is the contrastive objective used to fine-tune the video and text encoders.","marker":"[22]"}],"fun_headline_variants":["Masking static cues boosts zero-shot action recognition","Color-masking backgrounds and objects reduces static bias","Simple color masking helps zero-shot video models focus on actions","How masking scenes and objects sharpens zero-shot action models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that the segmentation masks (from GroundingDINO and SAM) correctly separate person, object, and background in every frame, so that masking removes static cues without also destroying the evidence the action actually needs.","fun_headline_variants_meta":{"raw":{"variants":["Masking static cues boosts zero-shot action recognition","Color-masking backgrounds and objects reduces static bias","Simple color masking helps zero-shot video models focus on actions","How masking scenes and objects sharpens zero-shot action models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1666,"prompt_tokens":941,"completion_tokens":725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":671}},"tokens_in":557,"tokens_out":725,"duration_ms":7245,"temperature":1.0,"reasoning_tokens":671,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:53:41.773416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: compute P-top1 and B-top1 after replacing masked regions with randomly sampled patches from other videos instead of uniform colors. If P-top1 gains vanish and B-top1 no longer drops, the reported effect is an artifact of color masking rather than a genuine reduction in static bias.","supporting_citations":[{"cited_title":"In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)","cited_arxiv_id":null,"evidence_quote":"ViFi-CLIP is one of the two base zero-shot action recognition models fine-tuned in the experiments."},{"cited_title":"International Journal of Computer Vision 129(5), 1675–1690 (2021)","cited_arxiv_id":null,"evidence_quote":"Mimetics is the weak-background-bias evaluation set central to showing that masking improves accuracy."},{"cited_title":"In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (Oct 2017)","cited_arxiv_id":null,"evidence_quote":"Something-Something v2 is the strong-object-bias benchmark used to evaluate object masking."},{"cited_title":"In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022)","cited_arxiv_id":null,"evidence_quote":"Chung et al. established static bias in fully-supervised action recognition and motivated the masking approach."}],"review_version":1}