{"id":"75687fb8-7be3-493f-b9bf-802fabecf177","arxiv_id":"2505.19503","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LAIN adds locality and interaction adapters to CLIP, reaching state-of-the-art zero-shot human-object interaction detection on HICO-DET and V-COCO.","lead":"This paper introduces LAIN, a system that improves zero-shot detection of human-object interactions by teaching CLIP to focus on local details and interaction patterns. It reports the best scores on widely used HOI benchmarks, beating previous CLIP-based methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on non-unified cross-paper baselines; no controlled re-runs or error bars are reported, so the claimed margin over prior work is not independently established.","rationale":"The reader correctly identifies baseline comparability as the main soft spot, and my read agrees. The architecture and ablations are internally coherent in design; I found no flaw in the locality/interaction adapter logic or the frozen-CLIP training scheme that would invalidate the method. The SOTA claim, however, is an external comparison claim, and the paper provides no controlled baseline runs. The potential protocol differences (DETR checkpoint, pair construction, text prompts, inference lambda, evaluation code) could each affect mAP at the scale of the reported margins. I also noticed a separate internal inconsistency worth flagging: under the UV setting, Table 3 reports LA+IA unseen mAP 30.50 while Table 6's adapter-position 1-12 row reports 28.96, and the latter exactly matches Table 1's LAIN UV unseen mAP (28.96). The ablation tables therefore appear to describe different configurations than the final model, or at least inconsistent numbers; this should be corrected or explained, though it does not by itself overturn the SOTA claim. Because the main unresolved issue is external verification, the conditional verdict is appropriate.","tokens_in":15445,"tokens_out":10571,"duration_ms":73371,"concrete_test":"Release LAIN code and re-run the closest competitors (at minimum CLIP4HOI and CMMP, ideally HOICLIP and BCOM) in a unified harness: same DETR detector and checkpoint, same detection confidence threshold, same HO-pair construction, same text prompt template, and same evaluation script on the five HICO-DET zero-shot splits. If, under identical conditions, LAIN's unseen-category mAP no longer exceeds every baseline on every setting, the 'significant margin' claim must be softened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.2 ('outperforming all previous methods by a significant margin under all zero-shot settings') is supported only by Table 1 numbers taken from prior publications. LAIN's pipeline has several free choices that can shift mAP: the DETR checkpoint and detection threshold, HO-pair construction (top-k or all pairs), the text prompt template ('A photo of a person [verb-ing] a [object].' plus learnable tokens), and the inference suppression in Eq. (17) with hyperparameter lambda. Unless CLIP4HOI, HOICLIP, CMMP, BCOM, and the 'CLIP' row are evaluated under exactly the same choices, the reported margins (e.g., +2.04 unseen UC over CLIP4HOI and +2.38 over CMMP) could be inflated by protocol differences rather than by locality and interaction awareness. The paper gives no code, no seeds, no error bars, and no statement that baselines were re-run in a unified harness. This does not make the method wrong, but it leaves the SOTA claim conditional on an unverified fairness assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LAIN, a zero-shot human-object interaction (HOI) detection framework built on a frozen CLIP visual encoder. LAIN inserts two lightweight adapters between CLIP layers: a Locality Adapter that aggregates neighboring patch information and spatial layout priors to capture fine-grained object details, and an Interaction Adapter that refines human-object tokens through cross-attention-based interaction pattern reasoning. The method is evaluated on HICO-DET and V-COCO under five zero-shot settings (UC, RF-UC, NF-UC, UV, UO) and fully supervised settings, reporting state-of-the-art results across all zero-shot settings. The paper also provides ablations showing that both adapters contribute, an analysis of adapter placement, a small-box analysis, and parameter/FLOP comparisons.","tokens_in":15703,"tokens_out":6787,"duration_ms":57591,"significance":"The central idea of injecting locality and interaction awareness into CLIP representations via parameter-efficient adapters is plausible and timely, addressing a known limitation of CLIP for region-level tasks. The extensive evaluation across multiple zero-shot settings and benchmarks, together with ablations and efficiency analysis, is a strength. However, the empirical claims are currently compromised by missing implementation details and uncontrolled comparisons to prior work; the reported margins are not independently verifiable. If the authors supply the missing hyperparameters, code, and a clearer baseline protocol, the contribution would be a solid step for zero-shot HOI detection. As written, the significance is conditional on these reproducibility and comparison issues being resolved.","major_comments":[{"comment":"The experimental section does not report critical hyperparameters: the inference suppression weight λ in Eq. (17), the adapter dimension D_a, the number of IPRM queries N_p, the convolutional kernel sizes K, the number of adapter layers L, the learning rate, batch size, number of epochs, and the number of random seeds. Without these values, the reported mAP numbers cannot be reproduced or compared against, and the claimed state-of-the-art result is not independently verifiable. This is a load-bearing issue because the method's contribution is empirical and the exact architecture choices (e.g., D_a, N_p, kernel sizes) are part of the method's design.","section":"Section 4.1, Equations (7), (13), (17)"},{"comment":"The claim that LAIN \"outperforms all previous methods by a significant margin under all zero-shot settings\" rests on Table 1 numbers taken from prior publications without a unified evaluation harness. The paper does not specify the DETR detector version, detection confidence thresholds, HO-pair construction strategy (top-k vs all valid pairs), text prompt template, or inference suppression settings (Eq. 17) used for the baseline rows. In particular, the CLIP baseline row, which is used in the introduction and Section 4.2 to argue that existing methods degrade CLIP's zero-shot performance, is not described at all. If the baselines used a different detector or prompt template, the reported margins (e.g., +2.04 UC unseen over CLIP4HOI, +2.38 over CMMP) could be inflated by protocol differences rather than by locality and interaction awareness. The authors should either re-run baselines under their own pipeline or clearly state the protocol used for each row and provide error bars over multiple runs.","section":"Section 4.2, Table 1"},{"comment":"The box-size analysis is presented as supporting evidence for the locality-awareness claim, but it is not a controlled comparison. The footnote states that ADA-CM's pretrained weights were obtained under \"UC-RF\" rather than UV, while the main ablation study is conducted under the UV setting; it is unclear under which zero-shot setting LAIN was trained for this table. If LAIN was trained under RF-UC (as the table caption suggests) while the ablation study uses UV, the comparison mixes training protocols and detector configurations. The authors need to clarify the settings and, ideally, compare all methods under the same setting with the same detected boxes so that the box-size gaps are attributable to the proposed modules rather than to protocol differences.","section":"Table 5 and Section 4.3"}],"minor_comments":[{"comment":"The first author's email address \"sanghuyn.kim\" appears to be a typo for \"sanghyun.kim\".","section":"Author affiliations"},{"comment":"The phrase \"we devise, LAIN,\" contains an unnecessary comma; consider rewording.","section":"Abstract"},{"comment":"In Eq. (7), γ_LA is defined as a learnable parameter in R^{D_clip}, but in Eq. (13) γ_IA is not explicitly typed. Please clarify whether γ_IA is a scalar or a vector in R^{D_clip}.","section":"Section 3.3, Equations (7) and (13)"},{"comment":"The footnote uses \"UC-RF\" interchangeably with \"RF-UC\"; these terms should be unified to avoid confusion.","section":"Table 5 footnote"},{"comment":"The paper does not explain how the CLIP (RF-UC) and CLIP (UV) numbers in Figure 1(c) and Table 1 are computed, including which detector and text template are used; adding this description would improve the clarity of the motivation and the comparison.","section":"Figure 1 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper comes from a strong group and addresses a timely problem. My main concern is that the empirical claims, which are the core contribution, are not yet reproducible or fully comparable to prior work. The lack of hyperparameter values and the absence of a described baseline protocol are fixable, but they are load-bearing for the state-of-the-art claim. If the authors provide code, detailed settings, and a careful baseline description (ideally re-running at least the closest baselines CLIP4HOI and CMMP under the same detector and inference protocol), I would be willing to reconsider. The box-size table also needs clarification. I do not see fundamental methodological flaws, but the manuscript in its current form is not fully ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LAIN is a clean, competent piece of applied CLIP adaptation: two small adapters—one for locality, one for interaction—inserted into frozen CLIP ViT layers. On HICO-DET it reports the best published zero-shot numbers across all five settings, and on V-COCO it's competitive if not first. The paper's own ablations are the strongest part: both adapters individually help, removing the spatial layout or visual context hurts, and the small-box analysis nicely confirms the locality story. The method is also light: 3M trainable parameters and 110G FLOPs, under most prior CLIP-based approaches.\n\nThe soft spots are real but not fatal. The headline SOTA claim rests entirely on Table 1 numbers taken from previous papers. LAIN's own pipeline has several choices that can move mAP—DETR checkpoint, detection threshold, pair construction, the text prompt, the inference lambda in Eq. (17)—and none of the baselines were re-run under the same harness. There are no error bars, no seeds are reported, and hyperparameters like lambda, Da, Np, and the kernel sizes are missing from the text. So the exact margin over CLIP4HOI or CMMP is not independently established. That is a common situation in this subfield, and the inconsistency could also hurt LAIN, but it means the 'significant margin' claim should be read with caution. The good news is the ablations are all against the authors' own baseline, so the internal logic of the paper holds up.\n\nI come down on the positive side. The method is well-motivated, the experiments are extensive, and the central claim—that locality and interaction awareness improve zero-shot HOI detection with CLIP—is supported by the controlled ablations. Who is this for? People working on zero-shot HOI detection specifically, and more broadly anyone adapting CLIP to region-level tasks. I would send it to reviewers. The revision requests should be about code, hyperparameters, and ideally one controlled baseline re-run; those are concrete and addressable. I'd probably also want a statement about how the DETR detector and prompt template align with the baselines. This is not a desk-reject.","headline":"A well-executed adapter-based SOTA for zero-shot HOI detection, with the main caveat being unverified cross-paper baselines and missing hyperparameters.","tokens_in":16211,"tokens_out":2750,"would_cite":true,"duration_ms":69444,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LAIN adapts a frozen CLIP with locality and interaction adapters to detect unseen human-object interactions, reporting state-of-the-art zero-shot HOI detection on HICO-DET and V-COCO.","keywords":["zero-shot human-object interaction detection","CLIP adaptation","locality awareness","interaction reasoning","HICO-DET","V-COCO","vision-language models"],"falsifier":"Run LAIN and the strongest prior methods (for example CMMP and CLIP4HOI) in a single codebase with the same DETR detector, the same text prompt, and the same evaluation script on the HICO-DET UV and RF-UC settings; if LAIN no longer exceeds them on unseen-class mAP, the claimed state-of-the-art margin is an artifact of comparison conditions.","tokens_in":15262,"feed_emoji":"🤝","tokens_out":3982,"duration_ms":34012,"temperature":0.7,"pith_summary":"The paper claims that CLIP's image representations, while strong for whole-image recognition, miss the fine-grained local detail needed to tell interactions apart when adapted to human-object interaction (HOI) detection. To fix this, the authors build LAIN, which inserts two lightweight adapters—a Locality Adapter that aggregates neighboring image patches and spatial layout information, and an Interaction Adapter that reasons about how human and object regions relate—into a frozen CLIP. On HICO-DET and V-COCO, LAIN reports higher mean average precision than prior zero-shot HOI methods across all zero-shot settings, and also improves fully-supervised rare-class performance. If correct, this shows that a small, parameter-efficient adaptation can unlock CLIP's generalization for detecting interactions with objects and verbs never seen in training.","feed_headline":"Two CLIP adapters push zero-shot HOI detection past raw CLIP","feed_subtitle":"Adding local detail and interaction reasoning to a frozen CLIP beats prior zero-shot HOI methods on both benchmarks.","key_machinery":"The central mechanism is the pair of adapters, LA and IA, inserted before each layer of a frozen CLIP vision transformer. LA updates every image patch token with aggregated neighborhood context and a spatial-layout prior; IA updates each human-object token with interaction-pattern features produced by cross-attention between human and object region features, in a module the paper calls the Interaction Pattern Reasoning Module. The claim is that these two complementary injections let CLIP keep its global knowledge while gaining the local and relational detail needed to score unseen HOI categories.","core_discovery":"The paper's central claim is that injecting locality awareness and interaction awareness into CLIP's patch and human-object tokens lets CLIP transfer to zero-shot HOI detection without hurting its generalization—unlike prior CLIP-based HOI adapters, which in some settings scored below raw CLIP on unseen classes. The authors show that the Locality Adapter (multi-scale convolutions over patch tokens with a spatial layout embedding derived from detected boxes and object text embeddings) plus the Interaction Adapter (ROI-aligned human and object features refined by cross-attention interaction-pattern reasoning) raise unseen-class mAP well above both CLIP alone and existing CLIP-adapted zero-shot methods. The paper also reports that gains are largest on small human and object boxes, where local detail matters most.","pith_inferences":["The locality-and-interaction adapter pattern may transfer to other region-level CLIP tasks, such as open-vocabulary detection or segmentation, which also suffer from CLIP's global bias.","Because the paper compares against published numbers, a unified re-implementation of baselines with the same detector and prompt template might change the reported margins; a fair head-to-head is needed to confirm the magnitude of the improvement.","The spatial-layout embedding that encodes object text embeddings suggests the method partly works by aligning patch features to object semantics, a component that could be isolated and tested in future ablations."],"forward_implications":["Zero-shot HOI detection can exceed raw CLIP's own zero-shot performance, closing the domain gap that hurt earlier CLIP adapters.","A few million trainable parameters (3.0M) suffice to adapt CLIP for HOI, far fewer than methods that train heavy decoders.","Small-instance HOI detection benefits most from locality-aware adaptation, suggesting local detail is the bottleneck for those cases.","The same adapters improve fully-supervised HOI detection, especially on rare classes."],"supporting_citations":[{"why":"CLIP provides the frozen visual and text encoders that LAIN adapts and the text embeddings used for zero-shot classification.","marker":"[39]"},{"why":"CLIP4HOI is a prior CLIP-adaptation method that establishes the HO-token and text-template baseline LAIN compares against and extends.","marker":"[36]"},{"why":"HOICLIP is a prior CLIP-based zero-shot HOI method and a key baseline in all zero-shot settings.","marker":"[37]"},{"why":"CMMP is a prior zero-shot HOI method using conditional multi-modal prompts, used as a direct comparison and ablation counterpart.","marker":"[24]"},{"why":"ADA-CM is a prior CLIP-context method whose published numbers and pretrained weights are used for box-size ablations.","marker":"[23]"},{"why":"DETR is the off-the-shelf object detector providing boxes, confidence scores, and object features for HO token construction.","marker":"[1]"},{"why":"HICO-DET is the main benchmark dataset on which all zero-shot settings and fully-supervised results are reported.","marker":"[2]"},{"why":"V-COCO is the second benchmark dataset used to evaluate fully-supervised performance and role average precision.","marker":"[12]"}],"fun_headline_variants":["LAIN fuses locality and interaction into CLIP for zero-shot HOI","Zero-shot HOI gets a boost: CLIP with locality and interaction awareness","Two adapters inject local and interaction cues into CLIP for HOI","LAIN sharpens CLIP on small objects for zero-shot HOI detection","Locality-aware CLIP beats raw CLIP in zero-shot human-object interaction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported margins assume that prior methods' published results were produced under the same detector, text prompt template, and evaluation code as LAIN, since the paper does not run those baselines in its own pipeline.","fun_headline_variants_meta":{"raw":{"variants":["LAIN fuses locality and interaction into CLIP for zero-shot HOI","Zero-shot HOI gets a boost: CLIP with locality and interaction awareness","Two adapters inject local and interaction cues into CLIP for HOI","LAIN sharpens CLIP on small objects for zero-shot HOI detection","Locality-aware CLIP beats raw CLIP in zero-shot human-object interaction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000848,"raw_usage":{"total_tokens":3667,"prompt_tokens":903,"completion_tokens":2764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2664}},"tokens_in":519,"tokens_out":2764,"duration_ms":20375,"temperature":1.0,"reasoning_tokens":2664,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:12:29.413798+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LAIN and the strongest prior methods (for example CMMP and CLIP4HOI) in a single codebase with the same DETR detector, the same text prompt, and the same evaluation script on the HICO-DET UV and RF-UC settings; if LAIN no longer exceeds them on unseen-class mAP, the claimed state-of-the-art margin is an artifact of comparison conditions.","supporting_citations":[{"cited_title":"Explor- ing conditional multi-modal prompts for zero-shot hoi de- tection","cited_arxiv_id":null,"evidence_quote":"CMMP is a prior zero-shot HOI method using conditional multi-modal prompts, used as a direct comparison and ablation counterpart."},{"cited_title":"Learning transferable visual models from natural language supervi- sion","cited_arxiv_id":null,"evidence_quote":"CLIP provides the frozen visual and text encoders that LAIN adapts and the text embeddings used for zero-shot classification."},{"cited_title":"Clip4hoi: Towards adapting clip for practi- cal zero-shot hoi detection","cited_arxiv_id":null,"evidence_quote":"CLIP4HOI is a prior CLIP-adaptation method that establishes the HO-token and text-template baseline LAIN compares against and extends."},{"cited_title":"Hoiclip: Efficient knowledge transfer for hoi detection with vision-language models","cited_arxiv_id":null,"evidence_quote":"HOICLIP is a prior CLIP-based zero-shot HOI method and a key baseline in all zero-shot settings."},{"cited_title":"Efficient adaptive human-object interac- tion detection with concept-guided memory","cited_arxiv_id":null,"evidence_quote":"ADA-CM is a prior CLIP-context method whose published numbers and pretrained weights are used for box-size ablations."},{"cited_title":"End-to- end object detection with transformers","cited_arxiv_id":null,"evidence_quote":"DETR is the off-the-shelf object detector providing boxes, confidence scores, and object features for HO token construction."},{"cited_title":"Learning to detect human-object interactions","cited_arxiv_id":null,"evidence_quote":"HICO-DET is the main benchmark dataset on which all zero-shot settings and fully-supervised results are reported."}],"review_version":1}