{"id":"e8db85cf-ab85-4edc-810b-4e8cf216e0be","arxiv_id":"2507.15542","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"HOLa achieves state-of-the-art zero-shot human-object interaction detection on HICO-DET by low-rank decomposing VLM text features and using LLM-generated action descriptions to regularize weight adaptation.","lead":"HOLa is a new method for zero-shot human-object interaction detection that decomposes vision-language model text features into shared basis features and per-class weights, then uses LLM-generated action descriptions to regularize the weights. It reports state-of-the-art results on the HICO-DET benchmark, including an unseen-class mAP of 27.91 in the unseen-verb setting.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5)'s KL regularization is dimensionally ill-specified as written, and the LLM-to-visual action alignment it presupposes is unvalidated; the SOTA claim leans on this module's +1.22 mAP gain.","rationale":"The reader's weakest assumption (LLM action descriptions may be misaligned with visual action semantics) points at the same component I find most fragile, so my concern is partially overlapping. However, I identify a more concrete, prior problem: Eq. (5) as written is dimensionally incoherent, because W has rows indexed by HOI classes but I selects columns of B, and the extracted 'i-th column of the adapted matrix W' cannot be KL-aligned with W_a, whose rows are action classes. This means the central regularization mechanism is not reproducible from the paper alone, independent of any semantic-alignment question. The ablations confirm the module is load-bearing: without LLM regularization (Table 5, Row 3), unseen mAP is 25.47, below the prior CMMP result of 26.23, so the SOTA claim depends on this term. Table 7's finding that LLM descriptions alone add only 0.37 mAP strengthens the need to verify the regularization itself rather than the descriptions. The supplementary C.1 note that HOI descriptions come from EZ-HOI/LLaVA rather than the LLM cited in Sec. 3.1 is a secondary inconsistency but worth flagging per the review rule. I do not think this changes the verdict: CONDITIONAL remains appropriate, because the released code may implement a correct variant and the issue may be a presentation error. The proposed concrete test (code inspection plus a zeroed-KL ablation) would settle whether the concern lands or reduces to a notational fix.","tokens_in":23851,"tokens_out":9071,"duration_ms":96118,"concrete_test":"Inspect the official HOLa code at github.com/ChelsieLei/HOLa for the forward pass of Eq. (5). Verify the shapes of War and Wa before the KL loss and how War is selected (rows of W indexed by HOI classes vs. columns indexed by basis). If the code defines a valid alignment, re-run Table 5 rows 3 vs 4 with the KL loss zeroed; the unseen-verb mAP should drop by the stated 1.22 if the mechanism is real. If the code reproduces the paper's dimensional mismatch, the ablation and SOTA numbers cannot be attributed to this term.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing component for the unseen-verb SOTA claim is the LLM-derived action regularization (Sec. 3.2, Eqs. 4-6): Table 5 shows it adds +1.22 mAP on unseen classes, the largest single-module gain, and without it HOLa (25.47) does not beat CMMP (26.23). As written, however, Eq. (5) is not a well-defined objective. W is N×m with rows indexed by HOI classes; W_a is N_a×k with rows indexed by actions; the index set I = {i | b_i ∈ B_a} selects columns of B, yet the paper extracts War as 'the i-th column of the adapted matrix W' and aligns it with W_a via DKL. This matches an N-dimensional column against a k-dimensional action row (or a matrix of the wrong shape); no normalization to a common probability space is specified. Thus the stated mechanism cannot be executed from the text, and the reported +1.22 mAP cannot be attributed to the described regularization without the code. Even under a charitable row-wise reinterpretation, the paper supplies no direct evidence that CLIP text encodings of LLaMA-3 action descriptions (which, per the supplementary prompt, list object names like 'car, dog, potted plant, elephant') align with visual action semantics; the action text may encode object co-occurrence rather than action. The only provided visualization (Fig. 8) shows weight similarities before/after adaptation, not alignment with W_a. The supplementary's C.1 note that HOI descriptions are taken from EZ-HOI/LLaVA rather than the LLaMA-3 LLM cited in Sec. 3.1 also muddies which text features enter the decomposition, though this is secondary.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HOLa, a zero-shot HOI detection method that decomposes VLM text features of HOI class descriptions into a low-rank class-shared basis B and per-class weights W, adapts W through a text adapter, regularizes the adapted weights with LLM-generated action descriptions via KL divergence, fuses action and object text features, and enriches visual features with human-object tokens. The method is evaluated on HICO-DET under unseen-verb, unseen-composition, and unseen-object zero-shot settings, reporting an unseen-verb mAP of 27.91 and claiming new state-of-the-art results across all settings, with ablations attributing the gains to the proposed modules.","tokens_in":24264,"tokens_out":7860,"duration_ms":80038,"significance":"If the results hold, HOLa makes a strong empirical contribution: it improves unseen-class mAP substantially over prior zero-shot HOI methods (e.g., +1.68 over CMMP in the unseen-verb setting) while using only 4.0M trainable parameters and releasing code. The low-rank decomposition of VLM text features is a principled way to preserve class-shared information, and the LLM-derived action regularization is a novel mechanism for transferring external action knowledge. The paper includes extensive ablations, visualizations, and comparisons with recent methods, and it reports competitive fully-supervised results as a bonus. However, the central regularization term in Eq. (5) is not well specified, an inconsistency exists regarding which LLM generates the HOI descriptions, and the reported numbers lack error bars; these issues currently prevent the stated state-of-the-art claim from being fully reproducible and statistically supported.","major_comments":[{"comment":"The KL regularization in Eq. (5) is not executable as written. W is N×m, W_a is N_a×k, and the index set I = {i | b_i ∈ B_a} selects basis vectors; the text states that W_ar contains the i-th column of the adapted matrix W, which has dimension N (or m, depending on the row/column convention), while W_a rows have dimension k. DKL[W_ar ∥ W_a] therefore compares objects of incompatible shape, and no normalization to a common probability space is specified. This is load-bearing: Table 5 shows that LLM-derived action regularization contributes +1.22 unseen mAP and is required to surpass CMMP's 26.23 in the unseen-verb setting. Please specify the exact shapes, the row/column convention for B and W, how indices in I map to rows of W_a, and how the KL is computed (e.g., row-wise softmax over which axis).","section":"Sec. 3.2, Eq. (5)"},{"comment":"The notation for the second action-regularization term is ambiguous: W_a is first introduced as the factorized action weights in Eq. (4), but then 'adapted action weights W_a' are regularized toward W_a in Eq. (6). It is unclear which copy is trainable, which is the frozen target, and whether the KL is over rows, columns, or the full matrix. Please clarify the roles of the two W_a symbols and align the notation with Eq. (5).","section":"Sec. 3.2, Eq. (6)"},{"comment":"The mechanism assumes that CLIP text encodings of LLaMA-3 action descriptions are semantically aligned with visual action features, so that pushing W toward W_a improves unseen-action discrimination. The paper provides no direct evidence for this alignment; the Fig. 8 visualization shows weight similarities before and after adaptation but not similarity to W_a. Since the +1.22 mAP gain is the decisive margin, please include a control experiment (e.g., permuted action descriptions, generic descriptions, or random W_a targets) and a quantitative alignment measure between W_ar and W_a to demonstrate that the gain comes from action semantics rather than from generic regularization.","section":"Sec. 3.2, Table 5"},{"comment":"Section 3.1 states that an LLM [9] (LLaMA-3) generates detailed HOI class descriptions that are encoded into F, but Supplementary C.1 states that HOI class descriptions are taken from EZ-HOI and generated with LLaVA, while LLaMA-3-8B is used only for action descriptions and vision-branch prior knowledge. This contradiction affects the reproducibility of the main text and the attribution of the method's gains; specify exactly which model generates which set of descriptions.","section":"Sec. 3.1 vs. Supplementary C.1"},{"comment":"All quantitative results are single runs, and numerous hyperparameters (α, β1-β4, γ1-γ2, τ, m, k) are tuned on the test benchmark without a validation split or error bars. Given that several SOTA margins are small (e.g., Table 3 unseen +1.16 over CMMP and Table 1 seen +1.60 over EZ-HOI), report mean±std over at least three seeds and describe the hyperparameter selection protocol to substantiate the claimed improvements.","section":"Sec. 4, Tables 1-4"}],"minor_comments":[{"comment":"The implementation details and the inference paragraph refer to 'Eq. (13)' for the action score, but Eq. (13) is in the supplementary; the main-text score equation is Eq. (7) and the final HOI score is Eq. (10). Please fix the cross-references.","section":"Sec. 4, Implementation Details; Sec. 3.4"},{"comment":"B is defined as R^{d×m} with basis vectors as columns, yet b_i is called the i-th row of B; please clarify the indexing convention to avoid confusion with the subsequent column-based extraction of W.","section":"Sec. 3.2"},{"comment":"The text says HOLa exceeds CMMP by 1.59 mAP on unseen classes in the RF-UC setting, but Table 3 shows a difference of 1.16 (30.61 vs. 29.45); the 1.59 figure is the NF-UC margin and should be corrected.","section":"Sec. 4.1, RF-UC paragraph"},{"comment":"The main text refers to 'Table 15' for the VLM feature decomposition and adaptation ablation, but Table 15 appears only in the supplementary; renumber or explicitly state that the table is in the appendix.","section":"Sec. 4.2, Table 5 reference"},{"comment":"The supplementary reports that varying β1 to 0.5 or β2 to 0.5 yields higher unseen mAP (28.32 and 28.81) than the chosen configuration (27.91), yet the default is selected based on full/seen mAP; please state the selection criterion explicitly so the hyperparameter choice is transparent.","section":"Supplementary Table 16"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid application-oriented contribution with code and extensive ablations, and the low-rank decomposition of VLM text features plus LLM action regularization is a distinct enough extension over the authors' prior EZ-HOI work. The main risks are the ill-specified Eq. (5) and the absence of error bars and a validation protocol, which are fixable in revision. The inconsistency about which LLM generates the HOI descriptions should also be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: solid empirical paper, current SOTA on HICO-DET zero-shot HOI across all four settings, with clean ablations. The low-rank decomposition idea is genuinely new, and the gains look consistent. But the stress-test concern is real: Eq. (5) as written does not type-check, and the LLM-alignment assumption it rests on is unexamined. Until that is fixed, I would not fully trust the attribution of the +1.22 mAP gain.\n\nWhat is good: factorizing VLM HOI text features into class-shared basis B and per-class weights W, adapting only W, and regularizing an action-related subset with weights factorized from LLM action descriptions is a sensible way to inject unseen-class information without retraining the text encoder. Tables 1–4 are consistent across settings; the ablations are thorough (feature decomposition, weight adaptation, LLM reg, text fusion, HO tokens, plus rank and loss-weight studies in the supplement); and the weight-similarity visualization supports the claim that action clustering improves. Code is promised. I believe this is a genuine empirical advance.\n\nWhere it is soft: the biggest issue is the one the stress-test flags. In Sec. 3.2, War is extracted as columns of the adapted W ('the i-th column'), while Wa is N_a×k with rows indexed by actions. Those shapes do not match, and no normalization to a common probability space is given for the KL. So the stated mechanism cannot be executed from the text. The ablation shows this module adds 1.22 mAP unseen, the difference between beating CMMP and not, so the headline claim leans on an underspecified step. This is fixable with a clearer definition and the code, but it is a real gap. Relatedly, the supplementary says HOI descriptions come from EZ-HOI/LLaVA while Sec. 3.1 cites LLaMA-3; that inconsistency should be resolved. The whole LLM-regularization idea also presupposes that CLIP text encodings of action descriptions align with visual action semantics; the prompt shown lists object names, so the 'action' text may partly encode object co-occurrence. Only end-task ablations are offered, no direct alignment analysis. Minor points: no error bars, and several hyperparameters (α, β's, τ, rank) are tuned on the benchmark without a described held-out protocol. None of this makes me think the results are fabricated; it makes me think the paper needs a revision that specifies the mechanism and validates the alignment.\n\nBottom line: if the code matches the intended mechanism, this is a genuine SOTA advance in zero-shot HOI. As written, the key equation is too ambiguous to trust the attribution. Send it to review; a serious referee can untangle the shape issue and ask for the alignment analysis.","headline":"Solid new SOTA for zero-shot HOI with a genuinely novel low-rank decomposition, but the key KL regularization in Eq. (5) does not type-check as written and the LLM-alignment assumption is unexamined.","tokens_in":24820,"tokens_out":3777,"would_cite":true,"duration_ms":36415,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decomposing VLM text features into class-shared basis and class-specific weights, guided by LLM action descriptions, lifts zero-shot HOI detection to a new state of the art: 27.91 unseen-verb mAP on HICO-DET.","keywords":["zero-shot HOI detection","human-object interaction","vision-language models","low-rank feature decomposition","LLM-derived regularization","action distinction","HICO-DET","unseen action generalization"],"falsifier":"Swap the LLM-generated action descriptions for generic, action-free templates (e.g., \"a person doing something with an object\") while keeping every other module and loss fixed; if unseen-verb mAP stays near 27.91 instead of dropping, the KL regularization is not actually injecting action semantics, and the alignment premise of the method fails.","tokens_in":23615,"feed_emoji":"🎯","tokens_out":11494,"duration_ms":102418,"temperature":0.7,"pith_summary":"The paper tries to establish that zero-shot human-object interaction (HOI) detection — recognizing actions in images that never appeared during training — improves sharply when VLM text features are restructured rather than used directly. HOLa factorizes the text features of each HOI class into a class-shared basis and per-class weights, so the shared basis carries information that transfers to unseen classes while the per-class weights, adapted under supervision and steered by LLM-generated action descriptions, separate actions that share the same object. The paper reports a new state of the art across all four zero-shot HOI settings on HICO-DET, led by an unseen-class mAP of 27.91 in the unseen-verb setting. The central claim a sympathetic reader should take away is that the low-rank decomposition, not the VLM itself, is what unlocks unseen-action generalization.","feed_headline":"27.91 mAP: low-rank split lifts zero-shot action detection","feed_subtitle":"Splitting VLM text features into shared basis and per-class weights lets models recognize actions never seen in training.","key_machinery":"The engine is the pair of factorization identities $F = WB^{\\top}$ for HOI text features and $F_a = W_a B_a^{\\top}$ for action text features, coupled by sharing the basis: $B_a$ is a subset sampled from $B$. The class-shared basis $B$, kept out of the classification loss and updated only by reconstruction, is what preserves unseen-class information; the adapted per-class weights $W$, regularized toward $W_a$ by the KL term $D_{\\text{KL}}[W_{ar} \\| W_a]$, are what sharpen same-object action distinction. On the vision side, human-object tokens $T_{ho} = (f_h + f_o)/2 + f_{\\text{spatial}}$ — mean human and object appearance features plus an MLP-encoded spatial vector — are pushed through the adapted VLM visual encoder so that interaction structure enters the similarity scores of Eq. (7).","core_discovery":"HOLa's central claim is that the tension in zero-shot HOI detection — generalizing to unseen classes while distinguishing actions on the same object — is resolved by writing VLM text features as a low-rank product $F \\approx WB^{\\top}$ and letting the two factors play different roles. The basis $B$ is shared by all HOI classes, is updated only by the reconstruction loss, and thus preserves the representation of unseen classes; the weights $W$ are per-class and are adapted by a text adapter under the action classification loss, letting them carry the details that separate \"ride bike\" from \"jump bike\". To stop this adaptation from overfitting seen actions, action text features are factorized into weights $W_a$ on a basis $B_a$ drawn from $B$, and the adapted HOI weights are pulled toward $W_a$ by KL divergence, injecting LLM-derived action knowledge into classes never annotated. In the vision branch, human-object tokens combining appearance and spatial cues enrich the visual encoder's interaction representation. The evidence offered is the HICO-DET tables: state-of-the-art unseen mAP in all four zero-shot settings, including 27.91 unseen-verb mAP, plus ablations attributing the gain to each component.","pith_inferences":["The decomposition-plus-coupling recipe is a candidate template for other fine-grained VLM tasks, such as attribute-object composition or fine-grained action recognition, where the same failure mode appears: VLM features under-differentiate categories that share an object or noun.","The paper never directly measures whether the action weights $W_a$ are semantically aligned with visual action features, so a useful extension is to probe that alignment; if it is weak, the KL term may be acting as a generic regularizer rather than a carrier of action semantics.","An adaptive rank per action or per HOI class, informed by the rank-ablation trade-off, could plausibly push unseen-verb mAP above the fixed rank $m = 71$ used here.","The authors themselves flag that HOLa depends on predefined unseen class names; that constraint makes open-vocabulary HOI — where the class set is not fixed at training time — the natural stress test of whether the decomposition generalizes beyond enumerated names."],"forward_implications":["If the central claim holds, zero-shot HOI methods built on frozen VLM feature spaces have a large margin to recover: HOLa reports a +6.40 gain in unseen-verb mAP over the previous best.","The rank of the decomposition becomes a practical trade-off knob: lower ranks (e.g., 17) emphasize class-shared information and raise unseen mAP, while higher ranks (e.g., 119) retain more class detail and raise seen mAP.","The design carries over to fully supervised HOI detection, where the paper reports 35.41 mAP on HICO-DET with a ViT-B backbone and 39.05 mAP with ViT-L, above existing two-stage HOI methods.","The method needs only class names and LLM descriptions, not visual data of unseen classes, during training — a practical property when collecting interaction annotations is expensive."],"supporting_citations":[{"why":"CLIP — the vision-language model whose text and visual encoders produce all features that HOLa decomposes and adapts.","marker":"[42]"},{"why":"HICO-DET — the dataset and zero-shot evaluation protocol on which every comparison and claim rests.","marker":"[5]"},{"why":"Llama 3 — the LLM that generates the action descriptions behind the action regularization and the interaction prior knowledge.","marker":"[9]"},{"why":"EZ-HOI — source of the LLM-generated HOI class descriptions that are encoded into the text features F being decomposed.","marker":"[22]"},{"why":"CMMP — the previous best in the unseen-verb setting that HOLa claims to surpass in all four zero-shot settings.","marker":"[26]"},{"why":"HOICLIP — the frozen-VLM baseline the paper contrasts in Fig. 1 and compares against in the action-dissimilarity plot of Fig. 5.","marker":"[39]"},{"why":"ADA-CM — supplies the adapter design inserted into the visual encoder and serves as a baseline in every zero-shot table.","marker":"[24]"},{"why":"DETR — the object detector that produces the human and object detections on which the two-stage pipeline operates.","marker":"[4]"}],"fun_headline_variants":["Low-rank VLM split boosts unseen action mAP to 27.91","HOLa: shared basis plus per-class weights hits 27.91 unseen mAP","Zero-shot HOI: decomposing text features yields 27.91 unseen mAP","Low-rank feature factorization lifts zero-shot HOI to 27.91 mAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM's action descriptions, once encoded by the VLM and factorized into $W_a$, capture action structure that matches the visual encoder's notion of those actions; if the descriptions are generic, object-biased, or misaligned with visual features, the KL regularization pulls the adapted weights in a direction that does not help unseen actions, and the paper only checks this indirectly through end-task ablations.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank VLM split boosts unseen action mAP to 27.91","HOLa: shared basis plus per-class weights hits 27.91 unseen mAP","Zero-shot HOI: decomposing text features yields 27.91 unseen mAP","Low-rank feature factorization lifts zero-shot HOI to 27.91 mAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1446,"prompt_tokens":1046,"completion_tokens":400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":662,"tokens_out":400,"duration_ms":4363,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:29:23.884185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Swap the LLM-generated action descriptions for generic, action-free templates (e.g., \"a person doing something with an object\") while keeping every other module and loss fixed; if unseen-verb mAP stays near 27.91 instead of dropping, the KL regularization is not actually injecting action semantics, and the alignment premise of the method fails.","supporting_citations":[{"cited_title":"Learning to detect human-object interactions","cited_arxiv_id":null,"evidence_quote":"HICO-DET — the dataset and zero-shot evaluation protocol on which every comparison and claim rests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EZ-HOI — source of the LLM-generated HOI class descriptions that are encoded into the text features F being decomposed."},{"cited_title":"Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection","cited_arxiv_id":null,"evidence_quote":"CMMP — the previous best in the unseen-verb setting that HOLa claims to surpass in all four zero-shot settings."},{"cited_title":"Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models","cited_arxiv_id":null,"evidence_quote":"HOICLIP — the frozen-VLM baseline the paper contrasts in Fig. 1 and compares against in the action-dissimilarity plot of Fig. 5."},{"cited_title":"Efficient adaptive human-object interac- tion detection with concept-guided memory","cited_arxiv_id":null,"evidence_quote":"ADA-CM — supplies the adapter design inserted into the visual encoder and serves as a baseline in every zero-shot table."},{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"DETR — the object detector that produces the human and object detections on which the two-stage pipeline operates."}],"review_version":1}