{"id":"d9606e8a-bbc1-4346-b53f-884b761a670d","arxiv_id":"2507.06510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A bilateral collaboration between an HOI detector and BLIP-2, combining attention bias guidance and token-level LLM supervision, achieves state-of-the-art open-vocabulary and closed-set HOI detection.","lead":"This paper introduces BC-HOI, a framework for open-vocabulary human-object interaction detection that has the detector guide a vision-language model to focus on specific interaction regions, while the VLM's language component supplies fine-grained caption-based supervision in return. If the reported results reproduce, the method delivers substantial accuracy gains over prior work on standard HOI benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The un-filtered GPT-4o captions used by LSG may leak held-out verb/object words into training, confounding the open-vocabulary evaluation.","rationale":"The reader's concerns about single-run results, error bars, and released code are valid but secondary. The central claim of the paper is that BC-HOI achieves state-of-the-art open-vocabulary HOI detection. That claim rests on the evaluation protocol in Section 4.2, which requires that unseen categories appear nowhere in training supervision. The LSG component, however, adds supervision from free-form GPT-4o captions without any control for whether those captions mention the very categories that are supposed to be unseen. Since the captioning prompt is explicitly designed to elicit interaction descriptions, describing a held-out interaction in a training image is not a rare edge case but a structural property of the setup. If even a modest fraction of training captions mention held-out categories, the reported improvements on Unseen and Full mAP are no longer a measure of compositional generalization. The proposed test is straightforward and could be run by the authors with their existing pipeline; if the overlap is zero, the concern is resolved, and the reader's conditional acceptance would be justified. Until then, the open-vocabulary results are not trustworthy, and the paper should be rejected or marked unverdictable in its current form. Because the closed-set results are less affected by this leak, the framework may still have merit, but the central claim as stated is unsupported.","tokens_in":21449,"tokens_out":8497,"duration_ms":93235,"concrete_test":"Reproduce the captioning step by running the paper's prompt (Supplementary Fig. A) with GPT-4o on the HICO-DET training set; parse the generated captions and compute the fraction of held-out verb or object tokens in each open-vocabulary split (RF-UC, NF-UC, UO, UV). If the overlap is nonzero, retrain BC-HOI with LSG applied only to captions whose tokens are all within the seen-category vocabulary (or with held-out tokens masked), and compare the Unseen/Full mAP with the reported numbers. A substantial drop would confirm caption leakage as the source of the open-vocabulary gains.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing concern is not the ABG chicken-and-egg dependency (which is partially acknowledged) but a potential test-set information leak in the open-vocabulary protocol. In Section 3.3, LSG trains the model to predict tokens of 'ground-truth' captions generated by GPT-4o from the training images (Eqs. 3-4), with higher weights on nouns and verbs. The captioning prompt in Supplementary Figure A explicitly asks for human-object interaction descriptions ('holding a leash and walking a dog'), so the captions will name objects and verbs. In the UC, UO, and UV evaluation splits of HICO-DET, certain verbs, objects, or combinations are withheld from the HOI training labels, but the corresponding interactions and objects still appear in the training images. Nothing in the paper filters or masks caption tokens that belong to held-out categories. If a training image contains an interaction with a held-out verb or object, GPT-4o will likely name it, and LSG will teach the HOI detector to associate those visual features with the same words used to build the text-based interaction classifiers. This provides direct supervision for unseen categories, invalidating the claim of open-vocabulary generalization and potentially explaining the large reported gains (e.g., +11.82 mAP on RF-UC unseen). The paper provides no analysis of overlap between the generated captions and the held-out category vocabulary.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BC-HOI, a one-stage open-vocabulary HOI detection framework that couples an HOI detector with BLIP-2 in a bilateral loop: (1) early fusion of CNN and ViT feature maps, (2) Attention Bias Guidance (ABG), which injects the detector's interaction-decoder cross-attention maps as bias into the VLM's self-attention and Q-Former to obtain instance-level interaction features, and (3) LLM-based Supervision Guidance (LSG), which trains the detector to produce attention maps that let the frozen OPT-2.7B predict token-level captions generated offline by GPT-4o. The authors report state-of-the-art results on HICO-DET and V-COCO in both closed and open-vocabulary settings, with the largest gains in the RF-UC unseen split (+11.82 mAP over the second-best method), and provide ablations of each component plus efficiency comparisons.","tokens_in":21751,"tokens_out":3960,"duration_ms":43458,"significance":"If the reported results hold, the bilateral-collaboration idea is a meaningful conceptual step: instead of treating the VLM as a fixed feature extractor, the detector's attention maps are used to make the VLM features instance-level, and the VLM's language head is used to provide fine-grained token-level supervision back to the detector. The paper is strong on experimental breadth: four open-vocabulary settings, closed settings on two benchmarks, and component ablations including attention-bias sources/destinations, caption sources, and token weights. The efficiency analysis is also useful. However, the central open-vocabulary claim is currently threatened by a potential test-set information leak in the LSG caption supervision, and the main comparisons are not fully controlled because the method receives an external GPT-4o caption signal that baselines do not. These issues must be resolved before the performance numbers can be taken as evidence for the proposed mechanism.","major_comments":[{"comment":"LSG trains the model to predict tokens of GPT-4o captions generated from the training images, and the captioning prompt in Supplementary Figure A explicitly requests human-object interaction descriptions (e.g., 'holding a leash and walking a dog'). In the UC, UO, and UV evaluation splits, held-out verbs, objects, and verb-object compositions can still appear in the training images, and nothing in the paper masks or filters caption tokens that belong to held-out categories. This means the model may receive direct token-level supervision for categories that are intended to be unseen, confounding the open-vocabulary evaluation and potentially explaining part of the large gains in Table 1 (e.g., RF-UC Unseen: 42.31 vs. 30.49 mAP for the second-best method). Please provide a quantitative analysis of the overlap between the caption vocabulary and the held-out category vocabulary for each split, and report results after excluding or masking all caption tokens belonging to held-out categories.","section":"Section 3.3, Eqs. (3)-(4); Supplementary Figure A"},{"comment":"The comparison in Table 1 is not fully controlled: BC-HOI is trained with an additional external supervision signal (GPT-4o captions) that none of the baselines receive. The 'w/o LSG' row in Table 7(e) is a useful internal control (NF-UC Unseen: 31.70 vs. 33.01), but the main open-vocabulary claims are made against methods without this extra signal. To separate the contribution of the bilateral collaboration from the contribution of the extra caption data, the paper should either include baselines trained with the same caption-supervision protocol or prominently report open-vocabulary numbers without LSG for the main results.","section":"Section 4.2, Table 1; Section 3.3"},{"comment":"ABG is the core novelty, but its benefit relies on the detector's cross-attention maps being reliable semantic guidance. The UO row of Table 1 shows that when the detector's attention degrades for novel objects, the gain on unseen categories is only +0.22 mAP (19.72 to 19.94), and the authors acknowledge this limitation. However, the paper also states in Section 4.2 that ABG 'provides accurate attention bias even for unseen verb-object combinations.' These two statements are in tension, and the paper does not provide an analysis of attention-map quality as a function of seen/unseen categories. Please add such an analysis or qualify the claim to the settings where it is supported by evidence.","section":"Section 4.4, Table 5; Table 7(c); Section 4.2"}],"minor_comments":[{"comment":"The hyperparameters α, β, and γ are fixed in Section 4.1, but Table 7(f) and Supplementary Table C show non-negligible sensitivity to these values (e.g., NF-UC Full ranges from 35.74 to 36.40 across weight settings). Reporting the chosen values in a table and discussing the sensitivity would improve reproducibility.","section":"Section 4.1 and Table 7(f)"},{"comment":"The table shows BC-HOI's inference model size as smaller than UniHOI's even though the training model is much larger; this works because the LLM is dropped at inference. Adding a footnote to that effect would prevent confusion.","section":"Table 4"},{"comment":"Reference [77] is cited for GPT-4o, but the entry points to an 'Improving language understanding by generative pre-training' URL for gpt-4; this citation appears incorrect and should be fixed.","section":"References"},{"comment":"The definition of ϕ as 'the Vision Tower' is introduced after Eq. (1), where a different symbol f^i is used for the interaction decoder; the notation is understandable but could be made more uniform to avoid ambiguity.","section":"Section 3.2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper's code is said to be released, but the provided URL is not included in the manuscript text (only in the abstract footnote). Given the strength of the SOTA claims and the potential caption-leak issue, I would encourage the editor to require the code or at least a detailed caption-generation protocol, including the exact prompts and any caption filtering steps, before accepting any revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The bilateral collaboration idea is genuinely new: feeding the HOI detector's cross-attention maps as bias into the VLM's self-attention (ABG) and using the LLM's token-level loss with part-of-speech weighting (LSG) is not something I have seen in HOI detection. The early fusion of CNN and VLM features is also a sensible addition. The ablations are thorough and show each piece contributes. If the results hold, this is a real step forward.\n\nBut I do not believe the open-vocabulary results are trustworthy as reported. The load-bearing problem is the LSG caption supervision. The paper uses GPT-4o to caption the training images, with a prompt that explicitly asks for human-object interaction descriptions. The LSG loss then trains the model to predict those caption tokens, with higher weights on nouns and verbs. In the UC/UO/UV splits, certain verbs, objects, and combinations are withheld from the HOI labels, but they still appear in the training images. Nothing in the paper filters or masks caption tokens that belong to held-out categories. So the detector is being directly supervised with the same words used to build the unseen classifiers. That could explain the very large gains, especially the +11.82 mAP on RF-UC unseen. The absence of any analysis of caption-to-held-out overlap is a serious omission.\n\nOther issues are smaller but worth noting: no code released, no error bars or multiple seeds, and the alpha, beta, gamma loss weights are tuned on the reported test setting. The external GPT-4o captions are also an extra supervision source that the baselines do not get, which makes the comparisons less clean.\n\nThe closed-set improvements are more believable because no label leakage is involved there, and the ablations are internally consistent. So I would not dismiss the paper. It deserves a serious referee, but the referee should demand a thorough leak analysis and ideally results with filtered captions. Without that, the open-vocabulary SOTA claim is not substantiated. If the authors can show the gains survive after masking held-out tokens, this becomes a strong paper. As it stands, the honest verdict is conditional: the mechanism is sound, the evidence is not fully there.","headline":"Strong engineering with plausible but unverified gains; the GPT-4o caption supervision may leak held-out categories and undermine the open-vocab claim.","tokens_in":625,"tokens_out":797,"would_cite":false,"duration_ms":27206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BC-HOI guides a vision-language model with the HOI detector's attention maps and feeds token-level LLM supervision back to the detector, achieving state-of-the-art results on HICO-DET and V-COCO in both open-vocabulary and closed settings.","keywords":["open vocabulary HOI detection","human-object interaction detection","vision-language models","attention bias guidance","BLIP-2","LLM-based supervision","DETR","HICO-DET"],"falsifier":"Measure the spatial concentration of the interaction-decoder cross-attention maps in the Unseen Object (UO) setting—for example, the entropy of each map or its intersection over union with the ground-truth human-object box—and compare it between queries for seen and unseen objects; the mechanism predicts diffuse maps for unseen objects accompanying the small +0.22 mAP gain, so equally concentrated maps would falsify the claim that attention-bias quality drives the result.","tokens_in":21268,"feed_emoji":"🎯","tokens_out":10094,"duration_ms":85013,"temperature":0.7,"pith_summary":"Open-vocabulary human-object interaction (HOI) detection asks a model to recognize <human, verb, object> triplets whose verb-object combinations, objects, or verbs were never seen in training. This paper argues that large vision-language models (VLMs) provide generalizable but spatially coarse features, which clashes with detection's need for instance-level precision. To resolve that, BC-HOI sets up a bilateral collaboration: the HOI detector's cross-attention maps are injected as bias into the VLM's attention layers, making the VLM produce fine-grained, instance-aligned features, and the VLM's language component in turn supervises the detector token-by-token using image captions, improving those attention maps. Experiments on HICO-DET and V-COCO report state-of-the-art results in both open-vocabulary and closed settings; for example, it exceeds the second-best method by 11.82 mAP on unseen categories in the rare-first unseen-composition setting (RF-UC), where rare verb-object pairs are held out during training. If confirmed, the work suggests that coarse VLM features can be made detection-grade by routing detector spatial knowledge into the VLM rather than only fusing outputs late.","feed_headline":"Two-way VLM feedback lifts unseen HOI mAP by 11.8 points","feed_subtitle":"Detector attention guides BLIP-2 to fine-grained features; LLM captions supervise back, lifting open-vocab HOI scores.","key_machinery":"The load-bearing object is the cross-attention map $\\mathbf{A}_{ho}\\in\\mathbb{R}^{N_q\\times h\\times w}$ produced by the interaction decoder via $(\\mathbf{E}_{ho},\\mathbf{A}_{ho})=f_i(\\mathbf{M},\\mathbf{E}_h\\oplus\\mathbf{E}_o)$, with each query's map treated as a spatial prior for one human-object pair. ABG up-samples, reshapes, and adapts each map through an FC layer, then adds it to the self-attention scores of the corresponding duplicated class tokens inside every layer of BLIP-2's ViT, and analogously biases the Q-Former's cross-attention with the same maps. A unified attention computation with mask matrices ($0$ bias for allowed query-key pairs and $-\\infty$ to exclude keys outside a query's receptive field) makes the bias injection fast and parallel. The opposite direction of the loop is carried by the auto-regressive loss $\\mathcal{L}_{\\mathrm{lsg}}=\\frac{\\gamma}{N_t}\\sum_{n=1}^{N_t} w_n\\,\\mathrm{CE}(t_n,g_n)$ over the LLM's caption tokens, with $w_n=\\alpha,\\beta,1$ for nouns, verbs, and other tokens; because BLIP-2 is frozen except for the new Q-Former queries $\\mathbf{Q}^q_f$, this loss back-propagates into the HOI detector and shapes $\\mathbf{A}_{ho}$ into a better bias source.","core_discovery":"The central claim is that a one-stage HOI detector and a frozen large vision-language model can improve each other in a loop, and that this loop is what unlocks open-vocabulary generalization. The HOI detector's interaction-decoder cross-attention maps focus on individual human-object pairs; ABG duplicates the VLM's class token once per HOI query and adds each query's cross-attention map as bias into every self-attention layer of BLIP-2's ViT and into the Q-Former, so the VLM emits $N_q$ fine-grained, detector-aligned embeddings instead of one holistic embedding. LSG then adds extra queries and class tokens whose biased VLM embeddings are fed to the OPT-2.7B language model to predict captions (generated by GPT-4o) word by word, with nouns and verbs weighted more heavily; because the VLM is frozen, the resulting gradient trains the HOI detector to produce attention bias that supports caption prediction. The authors report consistent state-of-the-art results: 43.01 Full mAP on HICO-DET and 68.20 mAP#1role on V-COCO in the closed setting, and 42.31, 33.01, and 31.18 mAP on unseen categories in the RF-UC, NF-UC, and UV open-vocabulary settings respectively.","pith_inferences":["The same attention-bias mechanism could transfer to other coarse-to-fine open-vocabulary tasks, such as open-vocabulary segmentation or referring expression grounding, where a detector or proposal network's spatial maps could steer a frozen VLM's patch attention.","Because the Unseen Object (UO) setting shows only a small unseen gain (+0.22 mAP), the framework's benefit appears tied to how well the detector can localize the object; pre-training LSG on large-scale image-text pairs with richer object categories, as the authors conjecture, is a directly testable next step.","The bilateral loop is effectively a form of online spatial prompt learning, in which the attention bias acts as a query-conditional prompt that adapts VLM features per instance, suggesting connections to parameter-efficient VLM adaptation beyond HOI detection.","One could test the sensitivity of the mechanism by corrupting $\\mathbf{A}_{ho}$ at inference (blurring it or replacing it with uniform maps); if performance barely drops, the claimed causal role of attention bias would be weakened."],"forward_implications":["Open-vocabulary HOI detection improves substantially on unseen verb-object combinations without any new external training data, because the detector's spatial prior compensates for the VLM's coarse output.","Frozen VLMs can be repurposed for fine-grained detection by steering their attention with detector-derived bias, avoiding expensive fine-tuning of the entire VLM.","LLM token-level supervision from captions is a viable alternative to pseudo-label extraction: it improves the detector's attention maps without explicit triplet labeling.","The framework's inference cost stays close to the baseline because the LLM component is dropped at test time, leaving only the biased VLM encoder and the detector.","The 11.82 mAP gain on unseen categories in the RF-UC setting and the large rare-category gains in the closed setting (45.76 versus 39.91 Rare mAP) indicate the method especially helps under-represented interactions."],"supporting_citations":[{"why":"GEN-VLKT supplies the HOI detector structure, training losses, and the four open-vocabulary evaluation settings (RF-UC, NF-UC, UO, UV) that BC-HOI inherits.","marker":"[4]"},{"why":"UniHOI is the closest baseline, providing the BLIP-2 setup and implementation details that BC-HOI adopts and then improves upon by the largest margins.","marker":"[14]"},{"why":"BLIP-2 is the vision-language model whose ViT, Q-Former, and frozen OPT-2.7B components are steered by ABG and LSG; its coarse cls-token behavior motivates the approach.","marker":"[36]"},{"why":"DP-HOI held the previous best unseen-category mAP in the RF-UC setting (30.49), and the paper's 11.82 mAP margin over it is the headline open-vocabulary result.","marker":"[52]"},{"why":"OPT-2.7B is the frozen LLM that emits the token-level caption predictions for the LSG loss during training and is discarded at inference.","marker":"[75]"},{"why":"GPT-4o generates the fine-grained, locality-aware captions that serve as ground truth for the auto-regressive LSG loss; an ablation shows the caption source affects the gains.","marker":"[77]"}],"fun_headline_variants":["Bilateral VLM loop sharpens open-vocab HOI detection","Attention bias guides VLM to fine-grained HOI features","LLM captions supervise HOI detector for unseen interactions","Two-way VLM feedback boosts open-vocab HOI performance","Collaborative VLM and detector excel at unseen HOIs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that the HOI detector's cross-attention maps point at the right interaction areas even for verb-object combinations, objects, or verbs the detector has never seen, because those maps are what steer the VLM's features.","fun_headline_variants_meta":{"raw":{"variants":["Bilateral VLM loop sharpens open-vocab HOI detection","Attention bias guides VLM to fine-grained HOI features","LLM captions supervise HOI detector for unseen interactions","Two-way VLM feedback boosts open-vocab HOI performance","Collaborative VLM and detector excel at unseen HOIs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1963,"prompt_tokens":1061,"completion_tokens":902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":816}},"tokens_in":677,"tokens_out":902,"duration_ms":9530,"temperature":1.0,"reasoning_tokens":816,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:01:38.676414+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the spatial concentration of the interaction-decoder cross-attention maps in the Unseen Object (UO) setting—for example, the entropy of each map or its intersection over union with the ground-truth human-object box—and compare it between queries for seen and unseen objects; the mechanism predicts diffuse maps for unseen objects accompanying the small +0.22 mAP gain, so equally concentrated maps would falsify the claim that attention-bias quality drives the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UniHOI is the closest baseline, providing the BLIP-2 setup and implementation details that BC-HOI adopts and then improves upon by the largest margins."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLIP-2 is the vision-language model whose ViT, Q-Former, and frozen OPT-2.7B components are steered by ABG and LSG; its coarse cls-token behavior motivates the approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DP-HOI held the previous best unseen-category mAP in the RF-UC setting (30.49), and the paper's 11.82 mAP margin over it is the headline open-vocabulary result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o generates the fine-grained, locality-aware captions that serve as ground truth for the auto-regressive LSG loss; an ablation shows the caption source affects the gains."}],"review_version":1}