{"id":"e5e49cd2-2d7d-46ff-bebb-f6c392491736","arxiv_id":"2508.00085","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"State-of-the-art video action recognition models fail to transfer high-level motion concepts to unseen contexts, even when the new context is only a minor variation of the training data.","lead":"This paper tests whether video action recognition models can recognize a broad action like 'punching' when the specific context changes, such as from a punching bag to a person. It introduces three benchmarks and finds that 13 models consistently lose 20% or more accuracy on this kind of transfer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"K400-TA coarse classes mix motion with object/scene categories, so the reported drops may reflect context generalization failure rather than motion transfer failure.","rationale":"I read the paper's central claim as: action recognition models show large accuracy drops when high-level actions are tested in novel fine-grained contexts, indicating failure to transfer motion concepts. For this to be a statement about motion, the coarse labels must be motion concepts. The least secure condition is construct validity of the benchmarks: several K400-TA coarse groups are object/scene themes, and Syn-TA includes non-motion transformations. The paper's own ablations show context dependence, but do not isolate motion. The reader's label-leakage concern applies mainly to the fine-motion results (Tables 3, 9) and does not threaten the coarse drop that anchors the central claim. I therefore disagree with the ranking of concerns, though the overall conditional verdict remains appropriate: the empirical drop is real, but the motion-specific interpretation needs an additional control. The proposed optical-flow / annotation check would settle whether the drop is motion-specific. No change to the reader's verdict is needed.","tokens_in":25841,"tokens_out":17242,"duration_ms":171469,"concrete_test":"Rerun the coarse known/unknown protocol on SSv2-TA and K400-TA using motion-only inputs (e.g., dense optical flow or extracted keypoint trajectories) instead of RGB frames, with the same fine-tuned backbones and hyperparameters. If the average Dabs remains at the 20+ point level, the drop is genuinely about motion transfer; if it collapses below ~10 points, the RGB-level drop is driven by object/scene features, and the central 'motion transferability' claim needs to be downgraded to a context-generalization claim. As a complementary check, have independent annotators label each coarse class as single-motion vs. thematic and recompute the headline drops excluding thematic classes; if the average Dabs drops below 20 in K400-TA, the across-dataset claim depends on non-motion categories.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that models fail to transfer high-level motion concepts to novel contexts—requires that the benchmark's coarse classes are genuinely single motion concepts and that the known/unknown split changes only the fine context, leaving the motion itself invariant. This premise is least secure in Sec. 3.2 and the class lists in Sec. D.2. Many K400-TA coarse groups are thematic categories rather than motions: e.g., 'toddler interaction' contains baby waking up, carrying baby, and crawling baby; 'music without instrument' contains air drumming, beatboxing, and singing; 'gardening' contains arranging flowers, planting trees, and stomping grapes. Within these groups, the fine classes do not share a common high-level motion; they share an object or scenario. For such classes, an accuracy drop from Set 1 to Set 2 measures failure to generalize object/scene cues, not failure to transfer a motion concept. Even in Syn-TA, several coarse labels—'changing color', 'changing shape', 'cloning', 'teleporting'—are appearance/state transformations rather than motions in the physical sense. In addition, the protocol changes object shape, background, and fine-motion details simultaneously (Sec. 3.2), so the reported Dabs conflates multiple context factors. Section 4.2's statement that 'all models... show a large drop' and the conclusion that models are 'fundamentally context-bound rather than motion-bound' therefore overinterpret the evidence: the drop is real, but attributing it specifically to failure of motion transfer is not justified by the current benchmark design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a benchmark framework for evaluating motion transferability in video action recognition. It constructs three datasets—Syn-TA (synthetic, controlled), K400-TA, and SSv2-TA—each organized into a coarse-to-fine hierarchy, and splits the fine classes into two sets that share the same coarse actions but have disjoint fine contexts. The authors train uni-modal and multi-modal models on one set and evaluate on both known and unknown contexts, reporting large accuracy drops (typically Dabs of 20% or more) for coarse actions and even larger drops for fine actions. They further analyze how model size and background texture affect transferability, and propose a disentanglement strategy that adds a fine-motion branch to EZ-CLIP to improve coarse-motion transfer on temporal datasets. The central claim is that current action recognition models are fundamentally context-bound rather than motion-bound.","tokens_in":26123,"tokens_out":5687,"duration_ms":56768,"significance":"If the findings are valid, this work provides a useful diagnostic benchmark for a subtle and under-explored form of generalization: transferring a known high-level motion to a novel fine-grained context within the same dataset. The paper is strong in scope—it evaluates 13+ models on three datasets, provides full class lists and per-set results in the supplementary, and releases code and datasets. The synthetic Syn-TA dataset is a valuable controlled testbed that partially isolates motion from real-world scene biases. However, the central interpretation is currently overextended: several coarse classes are not pure motion concepts, the Syn-TA protocol changes multiple factors simultaneously, and the fine-motion evaluation may be contaminated by pretraining label leakage. These issues do not destroy the empirical measurement of a performance drop, but they do undermine the specific attribution of that drop to failure of motion transfer.","major_comments":[{"comment":"The coarse classes in K400-TA are not consistently high-level motions. For example, the group 'toddler interaction' contains 'baby waking up', 'carrying baby', and 'crawling baby'; 'music without instrument' contains 'air drumming', 'beatboxing', and 'singing'; 'gardening' contains 'arranging flowers', 'planting trees', and 'stomping grapes'. In these groups the fine classes do not share a common motion—they share an object or scenario. For such classes, the reported accuracy drop from Set 1 to Set 2 measures failure to generalize object/scene cues, not failure to transfer a motion concept. This directly weakens the central claim in §4.2 that 'all models ... fail to transfer high-level motion concepts to novel contexts.' The authors should restrict the benchmark to coarse groups whose members share a verb or genuine motion pattern, or analyze motion-pure and thematic groups separately.","section":"§3.2, §D.2 (K400-TA class lists)"},{"comment":"In Syn-TA, the two sets differ simultaneously in object shapes, background images, and fine-motion details ('Each of the two sets has unique background images and object shapes'). Consequently Dabs conflates low-level appearance changes with motion-transfer failure. The statement in §4.2 that Syn-TA 'purely tests a model's motion understanding' is not supported, because a model could fail due to unseen colors, shapes, or textures rather than due to an inability to transfer the motion concept. A cleaner protocol would vary only the fine-motion detail while keeping object geometry and background fixed, or would factorially manipulate each factor to separate its contribution.","section":"§3.2 (Syn-TA construction)"},{"comment":"The 'unknown fine motions' for multimodal models include labels such as 'punching person (boxing)' from K400-TA Set 2. These labels are drawn from Kinetics-400, whose class names are very likely present in the web-scale text corpora used to pretrain CLIP and its video variants. The paper does not check for such overlap or exclude memorized labels. If these text descriptions were already seen during pretraining, the high Dabs for FineMotion-Unknown measures recall of memorized text rather than transfer of motion understanding. The authors should verify the absence of the test labels from pretraining data or design a truly novel set of textual descriptions for the unknown fine classes.","section":"§3.3, Tables 3 and 9"},{"comment":"All experiments are reported as single runs without confidence intervals, error bars, or repeated seeds. The headline numbers—'typical Dabs of 20% or more', the model-size trends in Fig. 6, and the disentanglement gains in Table 5—are point estimates that could shift with random seed, training epochs, or hardware. This is particularly concerning for the disentanglement results, where some gains are tiny (e.g., K400-TA Set 1 HM 76.07 vs. 76.06 in Table 12) and could easily be within run-to-run variance. The authors should provide variance estimates over at least three seeds for the main tables and figures.","section":"§4.1, Tables 2–11"},{"comment":"The cross-dataset comparison of harmonic means is confounded by the different number of coarse classes: Syn-TA has 20, K400-TA has 41, and SSv2-TA has 26. Chance-level accuracy differs substantially across these datasets, so the statement that 'Syn-TA proves as challenging as real-world datasets' is not supported by the raw HM values. A chance-normalized metric (e.g., normalized accuracy) or an analysis with matched class counts is needed before drawing conclusions about the relative difficulty of the synthetic versus real datasets.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The paper consistently says '13 state-of-the-art models' in the abstract and introduction, but §4 evaluates 8 unimodal, 5 multimodal, and 3 domain-generalization methods—16 total. Please clarify whether the DG methods are counted separately.","section":"Abstract and §4"},{"comment":"The 'Set 1/Set 2' columns are ambiguous for the coarse-class row; please explicitly show that both sets have the same number of coarse classes, or use separate columns for Set 1 and Set 2.","section":"Table 1"},{"comment":"The bubble-size plots use model parameter counts, but no parameter table is provided. Please include a table listing the number of parameters for each model to make the size analysis reproducible.","section":"§4.2, Fig. 6 and Fig. 12"},{"comment":"Dabs and HM are both reported, but Drel (relative drop) appears only in the supplementary. Since Drel is more interpretable for percentage-based claims, please consider reporting it in the main tables or at least in the main text discussion.","section":"§3.4"},{"comment":"Reference [55] is cited to support the claim that humans can recognize 'punching' beyond 'punching bag', but the cited book appears to be about curriculum design rather than human action generalization. Please replace it with a more relevant citation from the cognitive or action-recognition literature.","section":"References"},{"comment":"The disentanglement architecture is described only in the supplementary; the main text should include at least the key details—the branching structure, the residual connection formula, and the training loss—since this is presented as one of the paper's contributions.","section":"§3.5, Fig. 14"},{"comment":"The caption refers to 'top' and 'bottom' rows, but the figure appears to have only two panels arranged left and right; please clarify the layout description.","section":"Figure 10 caption"}],"recommendation":"major_revision","confidential_remarks":"The benchmark idea is timely and the empirical effort is substantial, but the paper currently overinterprets the Dabs numbers as evidence of a fundamental 'context-bound rather than motion-bound' failure. The class-purity, factor-confounding, and pretraining-label-leakage issues all need to be addressed before the central claim can be trusted. I would encourage the editors to ask for a revised version that either narrows the conclusion or reworks the benchmark to isolate motion transfer more cleanly. The disentanglement contribution is small and would be better positioned as an initial exploration rather than a main contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline finding holds up: across 13 models and three datasets, the drop from known to unknown contexts is large and consistent, and the coarse-to-fine intra-dataset split is a genuinely new evaluation device. Syn-TA is a useful controlled resource, and the plain-background ablation gives clean evidence that models lean on texture.\n\nWhere the paper oversells is the word \"motion.\" For SSv2-TA and much of Syn-TA the coarse classes are real motions. But in K400-TA many coarse groups are thematic, not kinematic: \"toddler interaction\" (baby waking up, carrying baby, crawling baby), \"music without instrument\" (air drumming, beatboxing, singing), \"gardening\" (arranging flowers, planting trees, stomping grapes). For these, the fine classes don't share a movement; they share an object or setting, so the reported Dabs is a context-generalization gap, not a motion-transfer gap. The stress-test note is right, and Section 4.2's conclusion that models are \"fundamentally context-bound rather than motion-bound\" goes beyond what the benchmark shows. Either the K400-TA taxonomy needs to justify a shared motion in each group, or the claim should be weakened.\n\nOther soft spots: no repeated runs or error bars in Tables 2-5, so model-level comparisons and the model-size analysis in Fig. 6 are shakier than the prose suggests. The fine-motion evaluation for multimodal models likely suffers from label leakage: captions like \"punching person (boxing)\" are almost certainly in the web-scale text used for CLIP pretraining, so \"unknown\" fine classes are not truly unseen at the text level. That doesn't damage the coarse results, but it should be checked or acknowledged. The disentanglement section is a minor add-on; a few points of harmonic mean on one backbone does not strongly support the architectural claims.\n\nWhat is solid: the datasets and splits are released, the evaluation covers many models, and the core empirical pattern—models trained on one set of fine contexts degrade on disjoint fine contexts of the same coarse label—is robust. That is new and useful even if the clean \"motion understanding\" interpretation is not fully earned in K400-TA. This deserves a serious referee. I would send it out, with a request for class-taxonomy validation, error bars, and a label-leakage check. Reading group: maybe—mostly to argue about what the drop actually measures.","headline":"The drop-on-unknown-context finding is solid and worth publishing, but \"motion transferability\" overstates it for K400-TA, where several coarse groups share scenarios, not motions.","tokens_in":26644,"tokens_out":4663,"would_cite":true,"duration_ms":39825,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current action recognition models lose 20 or more accuracy points when a motion appears in a new context, even when the coarse action stays the same.","keywords":["action recognition","motion transferability","fine-grained actions","coarse-to-fine hierarchy","video understanding","zero-shot generalization","synthetic benchmark","disentanglement"],"falsifier":"Inspect the training corpora of CLIP or the video-language models for the exact unknown fine labels (for example, 'punching person (boxing)', 'cleaning gutters', 'poking a hole into something soft'). If those strings occur in pretraining text and removing or masking them during fine-tuning does not change the reported unknown-fine accuracy, the paper's conclusion that models genuinely understand the motion would be called into question.","tokens_in":25677,"feed_emoji":"🎬","tokens_out":6696,"duration_ms":64050,"temperature":0.7,"pith_summary":"This paper tries to establish that today's action recognition models learn contexts rather than motions: a model trained on videos of a punching bag does not transfer the broad action 'punching' to an unseen variant such as a person boxing. To test this, the authors build three datasets with a coarse-to-fine hierarchy (Syn-TA, K400-TA, SSv2-TA), split each into two sets with the same coarse actions but disjoint fine classes, and train 13 unimodal and multimodal models to recognize coarse actions in known versus unknown fine contexts. On every dataset and nearly every model, known-to-unknown accuracy drops by 20 points or more on average, and multimodal models drop even more on unknown fine actions. A bias-free synthetic dataset with 3D objects turns out to be the hardest, indicating that models lean on object and background cues rather than on motion itself. The paper also shows that feeding fine-grained context into a coarse branch during training improves coarse action transfer on temporally demanding datasets.","feed_headline":"Models fail at recognizing the same motion in a new context","feed_subtitle":"Benchmarks with coarse/fine splits show 20-point drops for all 13 tested models.","key_machinery":"The load-bearing machinery is the intra-dataset coarse-to-fine evaluation protocol: fine classes that share a coarse motion are split into Set 1 and Set 2, models train on one set and are tested on both known and unknown variants, forcing any correct coarse-action recognition to generalize across disjoint fine contexts. The second piece is a disentanglement head added to EZ-CLIP, duplicating the final transformer blocks into a coarse branch and a fine branch, with fine embeddings added residually to the coarse branch so the model is trained to predict both levels at once. The metrics are the absolute drop $D_{\\text{abs}}$ and the harmonic mean between known and unknown accuracies.","core_discovery":"The central claim is that high-level motion concepts do not transfer across contexts even within a single dataset. The authors formalize this with an intra-dataset coarse-to-fine hierarchy and find that every evaluated model shows a large absolute drop $D_{\\text{abs}}$ from known to unknown contexts, regardless of architecture, modality, or domain. They show that multimodal models with frozen CLIP weights sometimes do better on unknown fine classes than on the matching coarse classes because they can latch onto object words in fine captions. They also find that model size helps when spatial appearance dominates (K400-TA) but not when temporal reasoning dominates (SSv2-TA), and that removing background texture in Syn-TA improves results, supporting the context-reliance conclusion. Finally, they claim that a two-branch disentanglement head, with fine features injected into the coarse branch, improves coarse transferability on temporal datasets.","pith_inferences":["A practical consequence the authors do not spell out: deployment of action recognizers in novel environments (robots, surveillance, assistive AI) should expect large reliability gaps even when the activity category was seen during training.","The memorization caveat could be tested directly: search the web-scale text used to pretrain CLIP-style models for the exact fine labels used as unknown classes; if those strings appear, the fine-motion unknown results partly measure label recall rather than transfer.","The disentanglement result suggests a cheap intervention for existing video-language models: add a fine-grained auxiliary prediction head and keep it during inference, rather than fine-tuning with coarse labels only.","A natural extension the paper does not pursue is to use Syn-TA's controlled generation to vary one factor at a time (object shape, background, camera path, frame order) and pinpoint which manipulation causes the transfer collapse."],"forward_implications":["On standard benchmarks, reported accuracy overstates motion understanding, because models can solve familiar fine contexts without learning the underlying coarse motion.","Model scaling is not a cure for temporal transfer: increasing parameters helped spatial-cue datasets but not Something-Something-v2, so progress on motion transfer needs different architectural or training interventions.","Multimodal models that exploit object words in fine captions can appear to understand fine actions while failing the same coarse action, so fine-grained zero-shot numbers need context-normalized interpretation.","Using the Syn-TA realistic-versus-plain-background comparison, any future action model can be quickly diagnosed for background dependence before expensive video curation.","Training with a coarse-plus-fine objective, as the disentanglement head does, is a concrete recipe for improving coarse transfer on temporally difficult datasets."],"supporting_citations":[{"why":"Supplies the Kinetics400 source videos and class labels from which K400-TA is adapted.","marker":"[26]"},{"why":"Supplies the Something-something-v2 source videos whose class descriptions naturally yield the coarse/fine split for SSv2-TA.","marker":"[21]"},{"why":"Provides the 3D rendering tool used to generate the controlled Syn-TA videos.","marker":"[10]"},{"why":"Provides the CLIP visual and text embeddings that the five multimodal models build on.","marker":"[46]"},{"why":"EZ-CLIP is the base model for the proposed disentanglement architecture and is a strong multimodal baseline in the benchmark.","marker":"[1]"},{"why":"FROSTER is a multimodal baseline and reports the best coarse harmonic mean on K400-TA.","marker":"[25]"},{"why":"ViFi-CLIP is a multimodal baseline whose failure on unknown fine motions is analyzed in the paper.","marker":"[47]"},{"why":"X-CLIP is a multimodal baseline representing cross-frame prompting approaches.","marker":"[41]"},{"why":"ActionCLIP is a multimodal baseline and performs best on Syn-TA fine motions.","marker":"[57]"}],"fun_headline_variants":["Action models fail when motion context shifts","Same action, new context: models drop 20 points","Motion transfer gap: all 13 models stumble","Context change breaks action recognition models","Punching bag vs. person stumps action models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the assumption that the fine labels used as 'unknown' classes, such as 'punching person (boxing)', were not already memorized by the pretrained text encoders used in the multimodal models.","fun_headline_variants_meta":{"raw":{"variants":["Action models fail when motion context shifts","Same action, new context: models drop 20 points","Motion transfer gap: all 13 models stumble","Context change breaks action recognition models","Punching bag vs. person stumps action models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1444,"prompt_tokens":971,"completion_tokens":473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":587,"tokens_out":473,"duration_ms":4655,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:22:43.397369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the training corpora of CLIP or the video-language models for the exact unknown fine labels (for example, 'punching person (boxing)', 'cleaning gutters', 'poking a hole into something soft'). If those strings occur in pretraining text and removing or masking them during fine-tuning does not change the reported unknown-fine accuracy, the paper's conclusion that models genuinely understand the motion would be called into question.","supporting_citations":[{"cited_title":"The kinetics human action video dataset, 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the Kinetics400 source videos and class labels from which K400-TA is adapted."},{"cited_title":"Blender: A 3d modelling and rendering package","cited_arxiv_id":null,"evidence_quote":"Provides the 3D rendering tool used to generate the controlled Syn-TA videos."},{"cited_title":"Learning transferable visual models from natural language supervision, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP visual and text embeddings that the five multimodal models build on."},{"cited_title":"Ez- clip: Efficient zeroshot video action recognition, 2024","cited_arxiv_id":null,"evidence_quote":"EZ-CLIP is the base model for the proposed disentanglement architecture and is a strong multimodal baseline in the benchmark."},{"cited_title":"Froster: Frozen clip is a strong teacher for open-vocabulary action recognition, 2024","cited_arxiv_id":null,"evidence_quote":"FROSTER is a multimodal baseline and reports the best coarse harmonic mean on K400-TA."},{"cited_title":"Fine-tuned clip models are efficient video learners, 2023","cited_arxiv_id":null,"evidence_quote":"ViFi-CLIP is a multimodal baseline whose failure on unknown fine motions is analyzed in the paper."},{"cited_title":"Expanding language-image pretrained models for gen- eral video recognition, 2022","cited_arxiv_id":null,"evidence_quote":"X-CLIP is a multimodal baseline representing cross-frame prompting approaches."},{"cited_title":"Actionclip: A new paradigm for video action recognition, 2021","cited_arxiv_id":null,"evidence_quote":"ActionCLIP is a multimodal baseline and performs best on Syn-TA fine motions."}],"review_version":1}