{"id":"7906ce10-51c2-4525-a338-e06b332df141","arxiv_id":"2411.10922","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"OpenMixer detects unseen actions in video by initializing box queries from VLM attention maps and fusing VLM semantics into temporal queries, beating baselines on J-HMDB and UCF101-24.","lead":"OpenMixer is a query-based model that detects and localizes human actions never seen during training by combining a frozen vision-language model with an end-to-end detection transformer. The paper introduces the open-vocabulary action detection task and benchmarks, reporting large gains over baselines on J-HMDB and UCF101-24.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The location prior in Sec. 3.2 assumes the argmax-class patch-text attention is spatially faithful for novel actions, but this is only tested under generic noise on one dataset, not by class-selection error or base/novel stratification.","rationale":"The reader's weakest_assumption identifies exactly the same technical point: the method's localization quality hinges on the VLM patch-text attention being a faithful spatial prior for novel actions when the text is selected by global video-text matching. I agree this is the most load-bearing concern because the S-OMB is one of the three core contributions, and its initialization is the only mechanism that transfers VLM localizability to the detector. The paper's own Table 9 is the closest evidence, but it tests noise from generic box sources (G.T., detection, random) rather than the specific failure mode of a wrong argmax text class, and it reports only J-HMDB aggregate numbers without base/novel stratification or recall analysis. The unexplained gap between the G.T. upper bound and the attention-based prior means the assumption could account for a meaningful share of the reported performance. However, the ablation in Table 3 shows S-OMB removal hurts novel mAP by only about 4.7 points on J-HMDB, and Table 9 shows the attention prior is far better than random, so the concern does not invalidate the method; it warrants an explicit test. Thus the reader's CONDITIONAL verdict stands, and no verdict adjustment is needed.","tokens_in":22499,"tokens_out":6990,"duration_ms":76040,"concrete_test":"Run OpenMixer's E2E pipeline on J-HMDB and UCF101-24 with the location prior computed under three text selections: (a) the current argmax over the open vocabulary, (b) an oracle using the ground-truth action text per video, and (c) a class-agnostic text like 'a person' or the mean text over all classes. Hold all other components fixed and report novel-class video mAP. If (b) or (c) substantially exceeds (a) by more than about 3 mAP, the argmax selection is the bottleneck and the claimed open-vocabulary localization is not robust to vocabulary ambiguity. Additionally, compute the recall of the top-N sampled centers against ground-truth boxes separately for base vs novel classes on both datasets; a large novel-vs-base recall gap would confirm the unfaithfulness for novel actions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central generalization claim depends on Sec. 3.2's prior-location mechanism: the model computes f_t = argmax over the open vocabulary of f_v^T f_t, forms the patch-text map S = V ⊗ f_t, reverses it to Ŝ = 1 - S, and samples the top-N box centers. For a video with multiple actors or a novel action, the holistically matched f_t may correspond to the dominant or a seen action rather than each actor's action; if so, Ŝ is the wrong semantic map and the S-OMB starts from incorrect centers, degrading class-agnostic localization. Table 9 only perturbs the prior with detection boxes and random boxes on J-HMDB; it does not isolate the failure mode where the argmax text is wrong, and it does not report prior recall separately for novel classes. The gap between the G.T. upper bound (91.19) and the attention prior (86.34) leaves about 5 mAP points unexplained, so the faithfulness assumption is load-bearing and currently under-tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OpenMixer, a query-based DETR-style architecture for open-vocabulary action detection (OVAD) built on a frozen video vision-language model (CLIP-ViP). The method exploits the VLM's semantics through a temporal OpenMixer block (T-OMB) and a dynamically fused alignment (DFA) module, and its localizability through a spatial OpenMixer block (S-OMB) that initializes person-box centers from a reversed patch-text attention map. Experiments on J-HMDB and UCF101-24 under three protocols (zero-shot recognition with zero-shot localization, zero-shot recognition with trainable localization, and end-to-end training) report large improvements over an STMixer baseline using the same frozen VLM, together with component ablations and additional split-level results in the appendix.","tokens_in":22689,"tokens_out":8179,"duration_ms":83008,"significance":"If the reported results are robust, this is a valuable contribution to open-vocabulary action detection. The paper demonstrates that a frozen video VLM can supply both semantic and spatial priors to a learnable DETR-style head, and it contributes new benchmark protocols and public splits. The manuscript has notable strengths: the authors release code, models, and splits; they report five random splits for both 50%-50% and 75%-25% seen/unseen partitions; they include a generalized zero-shot evaluation in the appendix; and their ablations (Tables 3-5, 9) are carefully designed with upper- and lower-bound location priors. The central claims, however, depend on the assumption that the argmax-class patch-text attention is a spatially faithful prior for novel actions, and on a single-split comparison in the main table; both require additional evidence before the state-of-the-art claim is fully established.","major_comments":[{"comment":"The claim that the argmax-class patch-text attention yields a faithful spatial prior for novel actions is central to the method, but the evidence in Table 9 does not isolate the failure mode. The gap between the ground-truth-prior upper bound (91.19) and the attention prior (86.34) is about 5 mAP points, and Table 9 reports no prior recall, no base/novel stratification, and no UCF101-24 results. I request an additional analysis: (i) report the recall of the top-N sampled centers against ground-truth boxes separately for base and novel classes; (ii) replace the global argmax class with the ground-truth class when constructing S to quantify how much of the gap is due to class-selection error versus patch-text spatial noise; and (iii) repeat the noise-level comparison on UCF101-24, including a multi-person subset. This is load-bearing because Table 3 shows that removing S-OMB reduces novel-class mAP from 82.33 to 77.64.","section":"Sec. 3.2, Table 9"},{"comment":"The main SOTA comparison in Table 1 is based on a single random class split and on an individual base/novel evaluation protocol, while Appendix Tables 11-14 show large split-to-split variation (e.g., novel mAP on UCF101-24 ranges from 28.55 to 61.96 under the 75%-25% split). No baseline is evaluated on the same alternative splits, so the reader cannot tell whether the reported improvements over STMixer are within the split noise. Please report the baselines on the same five 50%-50% splits (or at least provide standard deviations or error bars for Table 1), and consider making the generalized zero-shot protocol of Table 10 the primary evaluation in the main text, with individual testing as an auxiliary result.","section":"Sec. 4.1, Table 1, Appendix D"},{"comment":"The concurrent work [78] addresses the same OVAD task, and the paper does not provide an empirical comparison. The reasons given in Appendix F (no released code, different splits and metrics) are understandable, but the absence of any quantitative positioning relative to the only other same-task method weakens the 'performs the best over baselines' claim. I ask the authors to add a direct comparison under a shared protocol if the data of [78] become available, or at least to include a table that translates their results to the J-HMDB/UCF101-24 protocol used here, and to state clearly in the abstract and introduction that the comparison excludes [78].","section":"Sec. 2, Appendix F"},{"comment":"The fixed-equal-fusion baseline (lambda=0.5) yields 51.48 mean mAP, which is far below both lambda=0 (68.84) and lambda=1 (74.06). This non-monotonic behavior is not explained by the paper's narrative about adaptivity, and it raises the possibility of an optimization or scaling artifact (e.g., feature magnitudes, initialization, or loss weighting). Please provide an explanation, report the training curves or convergence, and verify the result across random seeds. This is important because the DFA module's dynamic fusion is a core contribution.","section":"Table 5"}],"minor_comments":[{"comment":"The operator symbol is used for both the inner product between fv and ft and between V and ft, which makes the equations ambiguous; please use a dot product or explicitly define the tensor contraction.","section":"Sec. 3.2"},{"comment":"The phrase 'more than 10% performance gain' should be 'more than 10 mAP points' for precision.","section":"Sec. 4.1"},{"comment":"There is a typo in 'the sate-of-the-art performance'; it should be 'state-of-the-art'.","section":"Introduction"},{"comment":"The caption does not state how many random splits were used or which split is reported; please reference Appendix D explicitly in the caption.","section":"Table 1 caption"},{"comment":"The phrase 'we emphasize on the adaptation' is ungrammatical; consider 'we emphasize adaptation'.","section":"Limitations and Future Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the action-detection and open-vocabulary communities, and the architectural contributions are coherent. The main concerns are the evidential basis for the location-prior faithfulness (Major Comment 1) and the robustness of the SOTA comparison (Major Comment 2). The anomaly in Table 5 (Major Comment 4) also needs resolution. I recommend major revision with a request for the specific analyses described above."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a credible piece of work. It defines open-vocabulary action detection as a distinct task, sets up benchmarks on J-HMDB and UCF101-24, and proposes OpenMixer, a DETR-style detector built on a frozen video VLM. The main idea worth stealing is the location prior: they compute patch-text attention in the frozen VLM, reverse it, and sample initial box centers from that map. It is cheap, structure-agnostic, and clearly helps. The gains over the STMixer baseline are large and consistent, and the ablations in Tables 3-5 and 9 actually justify the design choices. The dynamic fusion and the decoupled spatial/temporal queries are also well motivated. The paper is honest about its limits — it explicitly says AVA is left out and explains why comparison with concurrent work [78] is not feasible. That counts for something.\n\nNow the soft spots. The evaluation is narrow: two small, somewhat dated datasets, and the main table uses a favorable per-base/novel protocol without error bars. The appendix does provide multiple splits and generalized zero-shot results, which helps, but a larger dataset like AVA would be needed to convince me the approach scales. The missing empirical comparison with [78] is a real gap, even if the authors' reasons for not doing it are understandable.\n\nThe stress-test concern about the location prior is fair but not fatal. It is true that the prior uses the argmax class text for the whole video, which could be the wrong class for novel actions or multi-actor scenes, and Table 9 does not isolate that failure mode. However, the random-prior lower bound (54.15) still beats STMixer (49.16), so the method does not depend on the prior being perfect. The 5 mAP gap between the attention prior and the ground-truth upper bound is worth probing, but it is not a load-bearing flaw.\n\nAlso, the claim that OpenMixer works with 'any video VLM' is only tested on one true video VLM, CLIP-ViP, plus image CLIP, which performs much worse. That claim should be softened.\n\nOverall: this deserves a serious referee. It defines a useful task, ships a well-engineered method, and the core idea is borne out by the experiments. I would send it to peer review, ask the authors for an AVA result if feasible, a comparison with [78] if it can be done fairly, and error bars on the main numbers. The paper will be cited, and I would bring it up in a reading group.","headline":"A solid, well-ablated entry to open-vocabulary action detection; the VLM-attention location prior is the real idea, though the evaluation is narrow and the concurrent comparison is missing.","tokens_in":23286,"tokens_out":2375,"would_cite":true,"duration_ms":25387,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenMixer, a query-based detector built on a frozen video-language model, detects and names human actions never seen in training, beating all baselines on J-HMDB and UCF101-24.","keywords":["open-vocabulary action detection","video action detection","vision-language models","detection transformer","query-based detector","zero-shot action recognition","actor localization","prompt-based classification"],"falsifier":"Measure the localization quality of the prior directly: take a held-out video set with actor bounding-box annotations, pick novel (unseen) action classes, compute the reversed patch-text attention of the frozen VLM for each video, and count how often the top-100 sampled positions fall inside a ground-truth actor box. If the hit rate is at or below chance for clips with small actors or subtle motion, the location-prior assumption fails where the paper's novel-class gains are claimed, and the reported localization cannot be coming from seeds derived from that map.","tokens_in":22223,"feed_emoji":"🎬","tokens_out":11767,"duration_ms":101389,"temperature":0.7,"pith_summary":"Open-vocabulary action detection means finding and naming actions in video that a model never saw in training. The paper argues this is achievable without fine-tuning a large vision-language model: a DETR-style detector, OpenMixer, harvests two resources from a frozen video VLM (the localizability of its patch-text attention, which shows where actors are, and the semantics of its video-level feature, which carries what the action is). The spatial block samples initial box centers from the VLM attention map, the temporal block conditions query decoding on the video feature, and a dynamic fusion module blends pre-trained semantics into learnable queries matched against GPT-generated action descriptions. On J-HMDB the model reaches 86.34 mean video mAP, with 82.33 on novel actions; on UCF101-24 it reaches 47.71 mean, with 34.23 on novel actions, outperforming all baselines. If this holds, action detection in an open world no longer needs region-level pre-training or annotated supervision for every possible action.","feed_headline":"OpenMixer outscores baselines at detecting unseen video actions","feed_subtitle":"Seeding boxes from a frozen video-language model's attention, OpenMixer tops every baseline on seen and unseen classes.","key_machinery":"The load-bearing mechanism is the reversed patch-text attention map $\\hat{S} = 1 - V \\otimes f_t$: the video VLM's patch features $V$ are correlated with the text feature $f_t$ of the single class whose embedding most closely matches the global video feature $f_v$, and the map is reversed because CLIP-style attention pooling produces high patch-text similarity on background regions. Top-$\\hat{S}$ positions supply the initial box centers for the spatial queries, turning \"where is the actor?\" into a cheap read-out from a frozen model instead of a learned proposal stage. Two further mechanisms carry the recognition side: the temporal block broadcasts $f_v$ into the query-video mixing (an AdaMixer-style design) so decoding is conditioned on test-time video content rather than class text, and the DFA module learns a query-specific scalar $\\lambda$ to interpolate between the frozen video feature and the learnable query before cosine-similarity classification against GPT-generated action descriptions.","core_discovery":"The central claim is that a frozen video VLM contains two untapped resources (localizability and semantics) that together close the representation gap between video-level pre-training and region-level action detection. Concretely, OpenMixer uses the reversed patch-text correlation $\\hat{S} = 1 - V \\otimes f_t$, where $f_t$ is the text feature of the class most similar to the global video feature $f_v$, as a prior distribution over person locations, and samples the top-$N$ positions as initial box centers for the spatial OpenMixer Block (S-OMB). The temporal block (T-OMB) broadcasts the video-level feature into the query-video mixing; the dynamically fused alignment (DFA) learns a per-query weight $\\lambda$ to fuse the frozen VLM's video feature with learnable query features, then classifies by cosine similarity against GPT-generated action descriptions. With localization decoupled from recognition (the spatial queries never touch the classifier), the model localizes class-agnostically and recognizes both base and novel actions, reporting 86.34 mean video mAP on J-HMDB and 47.71 on UCF101-24 in the end-to-end setting, ahead of all baselines.","pith_inferences":["A corollary the authors leave implicit: because random location priors collapse mean mAP from 86.34 to 54.15 while ground-truth priors push it to 91.19, the method's ceiling is set by how faithfully the VLM attention marks actors (improving the prior via multi-frame aggregation, ensembling over prompt descriptions, or attention across ViT layers may pay off more than architectural changes).","The recipe of seeding proposals from VLM patch-text attention, conditioning decoding on the global video feature, and fusing pre-trained semantics with a per-query weight should transfer to other supervision-starved region-level video tasks, such as open-vocabulary video grounding or referring-expression localization.","A testable extension: building the location prior from an ensemble of prompt descriptions rather than the single best-matching class text could improve novel-class localization for small or partially occluded actors, where one class match is most fragile.","The paper states that the large-scale AVA dataset is excluded; all evidence rests on two small benchmarks, so whether the gains survive on dense, multi-person, long-duration video is an open question the authors flag for future work."],"forward_implications":["Open-vocabulary action detection can be built on a frozen video VLM with a light DETR head: no region-level pre-training, no VLM fine-tuning, and no label supervision for novel actions.","Localization and recognition should be decoupled: feeding class text or spatial queries into the classifier hurts both base and novel performance, so class-agnostic localization is the right design for open vocabularies.","Descriptive, LLM-generated action prompts improve not just recognition but also temporal localization (temporal mAP rises from 83.57 to 91.62 on J-HMDB), so text quality is part of the detection pipeline.","The choice of VLM matters more than the head: swapping the video-pretrained CLIP-ViP for image CLIP drops mean mAP from 86.34 to 71.60, so motion-aware pre-training is a precondition for the approach.","Gains persist under generalized zero-shot testing with the full vocabulary given (74.28 versus 36.26 for the STMixer baseline on J-HMDB), not just when base and novel classes are scored separately."],"supporting_citations":[{"why":"The frozen video VLM whose patch attention supplies the location prior and whose video and text features supply the semantics that the whole detector harvests.","marker":"[80]"},{"why":"The vision-language alignment basis of the approach; also the image-CLIP variant the paper swaps in to show video pre-training matters.","marker":"[51]"},{"why":"The one-stage action detector used as the main baseline and the source of the query-video mixing design OpenMixer adapts.","marker":"[76]"},{"why":"The AdaMixer Q-V mixing module reused inside the OpenMixer blocks for query-video interaction.","marker":"[15]"},{"why":"The DETR set-prediction formulation, Hungarian matching, and bipartite losses that the end-to-end training relies on.","marker":"[2]"},{"why":"The prompt-tuning baseline whose learned soft prompts are compared against the paper's GPT-generated prompts.","marker":"[96]"},{"why":"The class-conditioned open-vocabulary DETR whose text-conditioning strategy the temporal block ablates against and rejects.","marker":"[86]"},{"why":"Generates the multi-sentence visual action descriptions used for query-text alignment.","marker":"[47]"},{"why":"The region-prompting open-vocabulary detector whose two-stage training and representation-gap analysis motivate the single-stage frozen-VLM design.","marker":"[79]"}],"fun_headline_variants":["OpenMixer: VLM localizability boosts open-vocab action detection","Frozen VLM priors let OpenMixer spot novel actions","OpenMixer tops baselines for open-vocab video actions","Unseen actions detected: OpenMixer uses VLM semantics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen video-language model's patch-text attention, computed with the single action description that best matches the whole video, marks where the actors actually are for actions the model never trained on; if that attention points at the wrong locations, the box proposals start wrong and the class-agnostic localization cannot recover.","fun_headline_variants_meta":{"raw":{"variants":["OpenMixer: VLM localizability boosts open-vocab action detection","Frozen VLM priors let OpenMixer spot novel actions","OpenMixer tops baselines for open-vocab video actions","Unseen actions detected: OpenMixer uses VLM semantics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000431,"raw_usage":{"total_tokens":2254,"prompt_tokens":1056,"completion_tokens":1198,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":1122}},"tokens_in":672,"tokens_out":1198,"duration_ms":10452,"temperature":1.0,"reasoning_tokens":1122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:09:14.394575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the localization quality of the prior directly: take a held-out video set with actor bounding-box annotations, pick novel (unseen) action classes, compute the reversed patch-text attention of the frozen VLM for each video, and count how often the top-100 sampled positions fall inside a ground-truth actor box. If the hit rate is at or below chance for clips with small actors or subtle motion, the location-prior assumption fails where the paper's novel-class gains are claimed, and the reported localization cannot be coming from seeds derived from that map.","supporting_citations":[{"cited_title":"Clip-vip: Adapting pre- trained image-text model to video-language representation alignment","cited_arxiv_id":null,"evidence_quote":"The frozen video VLM whose patch attention supplies the location prior and whose video and text features supply the semantics that the whole detector harvests."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"The vision-language alignment basis of the approach; also the image-CLIP variant the paper swaps in to show video pre-training matters."},{"cited_title":"Stmixer: A one-stage sparse action detector","cited_arxiv_id":null,"evidence_quote":"The one-stage action detector used as the main baseline and the source of the query-video mixing design OpenMixer adapts."},{"cited_title":"Adamixer: A fast-converging query-based object detector","cited_arxiv_id":null,"evidence_quote":"The AdaMixer Q-V mixing module reused inside the OpenMixer blocks for query-video interaction."},{"cited_title":"Open-vocabulary detr with conditional matching","cited_arxiv_id":null,"evidence_quote":"The class-conditioned open-vocabulary DETR whose text-conditioning strategy the temporal block ablates against and rejects."},{"cited_title":"Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-matching","cited_arxiv_id":null,"evidence_quote":"The region-prompting open-vocabulary detector whose two-stage training and representation-gap analysis motivate the single-stage frozen-VLM design."}],"review_version":1}