Pith. sign in

REVIEW 4 major objections 4 minor 49 references

ART: Adaptive Relation Tuning for Generalized Relation Prediction

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ART claims that adaptive instruction tuning lets a vision-language model predict visual relations it never saw in training, beating mainstream scene-graph models and naive tuned baselines on mean recall across held-out datasets.

desk verdict Solid instruction-tuning recipe for VRD with consistent mR gains, but the proposed gR/gmR metric is coupled to the training sampler, so the headline generalization claim is weaker than it looks. read the letter →

arxiv 2507.23543 v2 pith:W6CST7XD submitted 2025-07-31 cs.CV cs.AI

Gopika Sudhakaran , Hikaru Shindo , Patrick Schramowski , Simone Schaub-Meyer , Kristian Kersting , Stefan Roth This is my paper
classification cs.CVcs.AI
keywords visualrelationdetectioninstructiontuningvision-languagemodelsadaptivesamplingclassificationsemanticsimilaritygeneralizationscenegraphgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that a vision-language model can be taught to predict visual relations it never saw during training, if relation detection is reframed as instruction tuning and the training instances are chosen adaptively rather than randomly or exhaustively. The authors propose ART, which converts relation-detection benchmarks into question-answer prompts and repeatedly moves the most informative samples into the training set: uncertain correct predictions, confident misses, and false positives that are semantically far from the ground truth. Tuned on about 12% of Visual Genome with this procedure, the model reports higher mean recall and generalized mean recall than tuning on the full dataset, on random subsets, or on balanced random subsets, and it produces relations such as “floating on” and “on back of” that were never annotated in training. A reader should care because mainstream relation detectors only recognize relations they were trained on, while handcrafted prompts for vision-language models fail on fine-grained or unseen relations; a model that infers relations it was never trained on would remove a real bottleneck in scene understanding.

What carries the argument

The carrying mechanism is an adaptive sampling loop wrapped around instruction tuning. Relation triplets are first turned into a question template that names a high-level relation category (“Is there a prominent spatial relation between...”), with one positive answer and one “counter-negative” answer drawn from a mutually exclusive category according to a hand-set exclusivity table among spatial, semantic, and possessive relations. Then, in each loop, a balanced initialization covers all predicates, per-predicate budgets are scaled by recall shortfall, and per-predicate thresholds are fit to the entropy $H$ and semantic-similarity $S$ distributions of true positives, false negatives, and false positives; samples are selected as high-entropy true positives, high- and low-entropy false negatives, and low-similarity false positives. The load-bearing part is the similarity measure $S$, a cosine similarity between predicted and ground-truth text embeddings: it decides which false positives are harmless paraphrases (“girl with hair” versus “girl has hair”) and which are genuine errors (“bag under table” versus “bag on table”), and the same $S$ with the same 0.95 threshold defines the paper's generalized recall metrics gR and gmR.

What would settle it

Swap the similarity measure S for an independent oracle and recompute both halves of the pipeline: have human raters (or a different text-embedding model with its own threshold) judge whether each false positive is a valid paraphrase of the ground truth, use that judgment to re-select ART's training samples and to recompute gR and gmR, and check whether ART still beats its strongest baselines. If the ordering flips, or if ART's chosen samples become indistinguishable from random selection, the reported generalized-recall advantage is an artifact of S's calibration. A cheaper probe is to rerun evaluation with the similarity threshold moved from 0.95 to 0.90 and 0.98 and measure how much ART's gR and gmR move relative to the baselines'.

Watch

Extended reading notes

Core claim

The central claim is that instruction tuning a vision-language model on a small, adaptively selected subset of a relation-detection dataset preserves and extends the model's generalization, so that it predicts relation concepts beyond the training vocabulary. Concretely, the paper shows that training on roughly 12% of Visual Genome data selected by ART yields higher mean recall (mR) and generalized mean recall (gmR) than training on the full dataset, on a random subset, or on a balanced random subset, across test sets of rising difficulty: GQA, Open Images v4, and Open Images v6. ART's model also predicts relations absent from training annotations, a capability the paper demonstrates is missing from mainstream scene-graph models, and these richer relations transfer downstream: scene graphs generated by ART resolve deictic segmentation prompts that ground-truth scene graphs miss. The authors frame the result as evidence that for imbalanced relation data, which samples you tune on matters more than how much you tune.

Load-bearing premise

Everything leans on one measure: an embedding-based semantic similarity score, cut at a 0.95 threshold that the paper says was chosen by qualitative inspection, is assumed to reliably tell harmless paraphrases of a relation (girl has hair versus girl with hair) apart from genuinely wrong relations (bag under table versus bag on table), and the same measure and threshold run both the training-sample selection and the headline generalized-recall numbers, so if that score is miscalibrated both sides of the argument drift together.

Editorial extensions

If this is right

  • A vision-language model tuned on roughly 12% of an imbalanced relation dataset, selected adaptively, outperforms the same model tuned on the full dataset, so sample selection can dominate data volume for this task.
  • Relation concepts never present in training annotations can be predicted, and this transfers to datasets with wholly new objects and predicates, which mainstream scene-graph models do not do.
  • Pairing each positive relation with one counter-negative from a mutually exclusive high-level category improves mean recall more than random negatives, and one counter-negative is better than several.
  • Adaptive, per-predicate thresholds for entropy and similarity beat any single fixed threshold, implying the selection rule should be customized per predicate.
  • The predicted relations carry into a downstream task: a neuro-symbolic segmentation model resolves deictic prompts that ground-truth scene graphs fail on, with near-perfect mAP when object boxes are provided.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same similarity measure $S$ and the same 0.95 threshold drive both training-sample selection and the headline generalized-recall metrics, so the reported gains in gR and gmR are partly self-referential: if $S$ misjudges paraphrase equivalence, the sampling and the evaluation share the same error. These numbers would be worth re-checking against an independent semantic-equivalence oracle.
  • The exclusivity table among spatial, semantic, and possessive relations is a domain heuristic; on datasets with a different predicate vocabulary the table would likely need re-derivation, and ART's edge could shrink if the presumed mutual exclusions do not hold there.
  • ART's output is often more specific than the annotation it is scored against (for example, “girl petting dog” versus “girl interacts with dog”), which suggests standard exact-match metrics under-reward the method; a metric that credits semantic entailment could change the apparent gap between ART and random-sampling baselines.
  • Since adaptive sampling roughly triples training time while leaving inference cost unchanged, the method's practical payoff depends on whether downstream applications extract enough value from fine-grained relations to justify the extra tuning cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ART, an instruction-tuning framework for visual relation classification. The authors convert Visual Genome relation annotations into an instruction-following format with high-level relation categories (spatial/semantic/possessive) and counter-negative mining, then fine-tune a VLM (InstructBLIP with Vicuna-7B or FlanT5-XL) on a small, adaptively selected subset of the training data. The adaptive sampler (Algorithm 1) combines balanced initialization with per-predicate sampling that uses entropy on true positives and false negatives and a semantic-similarity score S on false positives, with per-predicate adaptive thresholds. The model is trained on VG and evaluated on GQA and Open Images (v4/v6), reporting Recall, mean Recall, and newly proposed generalized recall metrics gR/gmR that credit false positives whose similarity to the ground truth exceeds 0.95. The paper also evaluates a downstream deictic segmentation task with DeiSAM. The central claim is that ART outperforms naive instruction-tuned VLMs and classical VRD models on mean recall and can predict unseen, semantically meaningful relations.

Significance. If the main claims are correct, the paper makes a useful contribution: it shows how adaptive instance selection can improve the rare-relation and cross-dataset behavior of VLM-based relation classifiers, and it backs the approach with a released codebase, thorough ablations (Tables 2 and 3), honest computational-cost reporting (Table 6, Fig. 7), and a concrete downstream application. The core mR improvements over balanced-random baselines, especially on OI-v4 (e.g., Vicuna-7B mR@20/50 of 26.0/26.2 vs. 11.8/12.4 for balanced random), are non-trivial and are not affected by the generalized metric. However, the paper's headline generalization evidence is substantially carried by the new gR/gmR metrics, and those metrics share their similarity function S and 0.95 threshold with the adaptive sampler that guides training; this coupling is a serious load-bearing weakness that needs to be resolved before the claims can be accepted at face value.

major comments (4)
  1. [Sec. 3.2 (Algorithm 1, Step 2.2) and Appendix B] The generalized metrics gR/gmR and the adaptive sampler are coupled through the same semantic similarity function S and the same 0.95 threshold. Algorithm 1, Line 29, samples only false positives with S(s) < tFP, i.e., it explicitly declines to correct high-similarity false positives, while Appendix B defines gR/gmR by re-labeling exactly those high-similarity false positives (S > 0.95) as correct. Baselines are not trained with this S and thus receive no such credit during training. Consequently, the reported gR/gmR gains for ART can be inflated purely by this metric-sampler alignment, independent of any genuine semantic improvement. I ask the authors to (i) report exact-match mR/R@k alongside gR/gmR in all headline tables (Tab. 1 already does, but the accompanying text should not give the generalized metrics equal weight without this caveat), and (ii) add an evaluation where the similarity re-labeling is performed with an independently calibrated embedding model and a threshold fixed before inspecting ART's outputs, to show the generalization claim survives a change of S.
  2. [Appendix B (On semantic similarity for evaluation)] The similarity measure S is not specified in a reproducible way. The paper only refers to 'cosine similarity-based approach' and says the 0.95 threshold was 'selected based on qualitative analysis'; no embedding model, version, or computation details are given, and no calibration or error analysis is provided. Since S appears on both the training side (Step 2.2) and the evaluation side (gR/gmR), the absence of details makes the generalized-recall numbers unverifiable and the threshold choice appear unprincipled. Please specify the embedding model, the exact per-triplet similarity computation, and provide a sensitivity analysis of gR/gmR and of the final model's mR to the threshold (e.g., 0.90, 0.93, 0.95, 0.97, 0.99).
  3. [Table 4 and Sec. 4.2 (Q5)] The downstream segmentation result 'DeiSAM + ART (with bbox)' reaching mAP 99.98 on VG and 97.96 on GQA is implausibly high for a real segmentation task, especially when 'DeiSAM + GT (with bbox)' reaches only 35.04 and 43.84. This suggests that the evaluation protocol or the bbox-conditioned setup inadvertently leaks the solution (for example, by using the bounding box as a seed mask or by computing mAP over an extremely small candidate set). Please clarify the exact protocol for the 'with bbox' condition, report per-image metrics and failure cases, and verify that the comparison with DeiSAM + GT is apples-to-apples.
  4. [Figs. 4/6, Sec. 4.2 (Q1-Q2), and Appendix H] The claim that ART 'can infer unseen relation concepts' is supported only by counts of unique and unseen predictions (Figs. 4/6) and by qualitative examples (Appendix H). No quantitative precision or recall is reported on the subset of test predicates that never appear in the VG training vocabulary. Please compute per-predicate mR or precision@k on the held-out predicate subset for ART and for all baselines, so that the zero-shot/unseen-concept capability is measured rather than merely illustrated.
minor comments (4)
  1. [Table 2] In the first row, both R@k and mR@k are reported as 37.5/38.0 and 37.5/38.0; identical values for these two metrics are suspicious and should be checked for a typo.
  2. [Figs. 4 and 6] The figures showing unique and unseen relation predictions lack readable axis labels and numerical values, making it impossible to extract the counts that the text refers to; please add labeled axes and, if possible, tabulate the counts.
  3. [Sec. 4.2 (Q1)] The sentence 'Vicuna-7B+ARToutperforms the strongest baseline' contains a missing space between 'ART' and 'outperforms'; please fix the typo.
  4. [Appendix G, Table 7] The first row of Table 7 changes both tFP and tFN from the midpoint, while the text says 'lower t and increase h from their midpoint values'; please clarify the naming so that 'Lower-t' refers to both low-similarity FP and low-entropy FN thresholds, or split the rows to show each variation separately.

Circularity Check

1 steps flagged · score 6.0 of 10

The generalized-recall metrics gR/gmR share their semantic-similarity function S and its 0.95 threshold with ART's adaptive sampler, so the headline generalization gains are partially aligned with the training objective by construction.

  1. self definitional [Sec. 4.1 (Evaluation protocol and metrics), Sec. 3.2 / Algorithm 1 Step 2.2, Appendix B]
    "To further assess generalization, we propose generalized Recall (gR@k) and mean generalized Recall (mgR@k), which treat false positives with high semantic similarity (measured by S from Sec. 3.2) as true positives."

    The same semantic similarity function S is used on both sides of the pipeline. In Algorithm 1 Step 2.2, ART deliberately does not sample false positives with S(s) > tFP because they 'retain meaningful semantics, making them less critical for correction.' In Appendix B, the evaluation thresholds S at 95% and counts exactly those high-similarity false positives as true positives. Thus ART is trained to preserve the very predictions that gR/gmR reward: high-similarity FPs are left uncorrected during training and then re-labeled as correct during evaluation. The gR/gmR advantage over baselines in Table 1 is therefore partly a self-consistency between the training objective and the evaluation metric, not an independent measurement of generalization to unseen concepts.

full rationale

ART's central claim of improving over baselines is partially supported by non-circular evidence: the exact-match mean recall (mR@k) numbers in Table 1 and the downstream segmentation results do not depend on the semantic similarity function S. However, the paper's own generalized metrics gR/gmR are defined using S from Sec. 3.2, and the adaptive sampler is explicitly designed to avoid correcting high-similarity false positives because they are considered semantically valid. The evaluation then counts those same high-similarity false positives as true positives. This couples the training selection criterion to the headline evaluation metric, so the gR/gmR gains are inflated by construction if S is at all permissive. The threshold 0.95 was selected by qualitative analysis rather than calibrated independently, and no error analysis of S is provided. This is a genuine partial circularity, but it does not extend to mR, which is an exact-match metric, nor does it rely on self-citation or imported uniqueness theorems. The self-cited DeiSAM and VETO works are used as a downstream application and a baseline, respectively, and are not load-bearing for the core derivation. Overall, the paper has independent content, but the specific generalized-recall evidence for 'unseen relation concepts' is partially self-consistent rather than independently verified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper does not rest on a mathematical derivation; it is an empirical method. The free parameters are manually chosen thresholds and budgets that control both the sampling algorithm and the evaluation metric. The axioms are domain assumptions about the instruction format, the exclusivity of relation categories, and the reliability of uncertainty and similarity estimates. No new physical or conceptual entities are postulated; the 'counter-negative' and 'instruction set' are data constructs, not theoretical entities.

free parameters (4)
  • z-score threshold z_init = 1.96
    Set to capture 95% of the data in the entropy/similarity distributions; sensitivity not explored, and it controls the initial threshold widths in Algorithm 1.
  • similarity threshold for gR/gmR = 0.95
    Chosen by qualitative analysis (Appendix B) to decide when a false positive is semantically equivalent to the ground truth; directly controls the magnitude of the reported gR/gmR scores.
  • training data fraction = 12% of VG training data
    Selected because mR@k saturates near this point (Fig. 5); affects all reported results and is a post hoc data-selection choice.
  • sampling budget per loop = 2% of training data per iteration
    Hyperparameter of the adaptive sampling loop; no justification is given for this value beyond reaching the chosen 12% total over a number of loops.
assumptions (4)
  • ad hoc to paper The exclusivity rules in Eq. (1): spatial vs. possessive and semantic vs. possessive are mutually exclusive, while semantic vs. spatial are not.
    This assumption determines which negatives are mined during instruction-set construction. If incorrect, negative labels are wrong and the model learns distorted relation semantics. The paper justifies it with reasoning about possession being static, but it is a heuristic without external validation.
  • domain assumption Beam search entropy, averaged over beams and positions, is a valid measure of model uncertainty for sample selection.
    Used to rank samples in the adaptive sampling algorithm. The paper does not validate this entropy estimate against other uncertainty measures such as Monte Carlo dropout or ensembles.
  • domain assumption The instruction template with a yes/no question about a specific relation reliably exposes the true relation in the image.
    All training data is converted into this format. The mapping from visual content to a textual yes/no answer may miss context, and the template itself is an unvalidated modeling choice.
  • domain assumption Validation-set recall per predicate is a reliable proxy for allocating sampling budgets in Algorithm 1.
    Step 1 of Algorithm 1 assumes that recall on the validation set reflects each predicate's need for additional supervision and that this transfers to the held-out test distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ART: Adaptive Relation Tuning for Generalized Relation Prediction." pith.science (2026). https://pith.science/paper/W6CST7XD

@misc{pith2026250723543,
  author       = {Pith},
  title        = {Pith review of: ART: Adaptive Relation Tuning for Generalized Relation Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6CST7XD}},
  note         = {Machine review of arXiv:2507.23543}
}
read the original abstract

Visual relation detection (VRD) is the task of identifying the relationships between objects in a scene. VRD models trained solely on relation detection data struggle to generalize beyond the relations on which they are trained. While prompt tuning has been used to adapt vision-language models (VLMs) for VRD, it uses handcrafted prompts and struggles with novel or complex relations. We argue that instruction tuning offers a more effective solution by fine-tuning VLMs on diverse instructional data. We thus introduce ART, an Adaptive Relation Tuning framework that adapts VLMs for VRD through instruction tuning and strategic instance selection. By converting VRD datasets into an instruction tuning format and employing an adaptive sampling algorithm, ART directs the VLM to focus on informative relations while maintaining generalizability. Specifically, we focus on the relation classification, where subject-object boxes are given and the model predicts the predicate between them. We tune on a held-in set and evaluate across multiple held-out datasets of varying complexity. Our approach strongly improves over its baselines and can infer unseen relation concepts, a capability absent in mainstream VRD methods. We demonstrate ART's practical value by using the predicted relations for segmenting complex scenes.

Figures

Figures reproduced from arXiv: 2507.23543 by the authors.

Figure 1
Figure 1. ART predicts detailed, context-rich relationships, en￾hancing downstream reasoning, e.g., for segmentation. From the input image, ART (trained on VG) predicts unseen and infor￾mative relations like floating on and on back of (highlighted in green). This richer relational context, facilitated by careful tun￾ing of a VLM, e.g., allows the DeiSAM [32] segmentation model, which relies on scene graphs for spatial reasoni… view at source ↗
Figure 2
Figure 2. Relation tuning format template. s (subject), o (object), and p (predicate/relation). 3.1. Relation-tuning data creation A key aspect of our approach is relation-tuning data con￾struction, which involves carefully crafting questions to capture fine-grained object relations. A simple transforma￾tion of the ground-truth relation triplets ⟨s, p, o⟩ into a ques￾tion format such as “Is there a relation between ⟨s⟩ and ⟨o… view at source ↗
Figure 3
Figure 3. ART Framework. To construct the relation-tuning dataset S, we create question-response prompts for each image region (e.g., 1 and 2 ), including both positive and negative examples, which we call instruction sets. Balanced sampling ensures that instruction sets span all predicate categories in S, forming the training data Strain. This training data is used to fine-tune the vision-language model (VLM), M, while unuse… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Comparison of unique relation predictions (left) and unseen relation predictions (right) for the ART (Vicuna-7B) model across different datasets. tuning a VLM on more data can lead to suboptimal results if the data distribution is biased. Comparing random to bal￾anced …
Figure 5
Figure 5. Figure 5: Training data subsampling analysis. We plot mR@50 for ART and Naive-RT (balanced random) as a function of the training data percentage used (y-axis) on Vicuna model variants. i [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Comparison of unique relation predictions (left) and unseen relation predictions (right) for the ART (FlanT5) model across different datasets. These sampling decisions are made adaptively per predi￾cate using dynamically computed thresholds (based on per￾predicate entr…
Figure 7
Figure 7. Figure 7: Trade-off between computational cost and predictive performance as a function of training data usage. F.3. On R@k and gR@k performance trade-offs While ART achieves strong generalization and diversity, it may show lower R@k and gR@k compared to ran￾dom baselines in cas…
Figure 8
Figure 8. Figure 8: ART can be used to label missing annotations and predict new unseen predicates. Segmentation results with textual prompts (top) using DeiSAM [32], which segments objects via reasoning on scene graphs. ART successfully detects new relations and improves the segmentation…
Figure 9
Figure 9. Figure 9: Comparison of ART and its baselines on the GQA dataset. ART predicts sensible spatial relations similar to the ground-truth annotation such as, water under sky, while also identifying more informative relations than the ground truth, such as water reflecting sky, boat …
Figure 10
Figure 10. Figure 10: Comparison of ART and its baselines on the GQA dataset. The ground truth only provides a spatial relation in between animal and water, while ART predicts the descriptive interaction swimming in. Informative relation predictions are highlighted in green, while those th…
Figure 11
Figure 11. Figure 11: Comparison of ART and its baselines on the OI-v4 dataset. In contrast to the provided less informative ground-truth relation interacts with in girl interacts with dog, which raises the question “What kind of interaction?”, ART provides a much clearer interpretation th…
Figure 12
Figure 12. Figure 12: Comparison of ART and its baselines on the OI-v4 dataset. The ground truth relation holds between man and beer leaves an open question “What he intends to do with the beer?”, while the prediction drinking made by ART gives more context and the ongoing action. The Naiv…
Figure 13
Figure 13. Figure 13: Comparison of ART and its baselines on the OI-v6 dataset. ART predicts the more informative relation paddling between the man and canoe while also identifying sensible spatial relation in. In contrast, although both Naive-RT (random) and (balanced random) make reasona…
Figure 14
Figure 14. Figure 14: Comparison of ART and its baselines on the OI-v6 dataset. While the ground-truth relation contains between the mug and beer merely indicates the presence of beer in the mug, the relation filled with, predicted by ART, provides more detail by suggesting that the mug is…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 41 canonical work pages

  1. [1]

    Flamingo: A visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikołaj Bi...

    work page 2022

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In ICCV, pages 2425–2433, 2015. 1

    work page 2015

  3. [3]

    A comprehensive survey of scene graphs: Generation and application

    Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiao- jiang Chen, and Alex Hauptmann. A comprehensive survey of scene graphs: Generation and application. IEEE Trans. Pattern Anal. Mach. Intell., 45(1):1–26, 2021. 1

    work page 2021

  4. [4]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. J. Mach. Learn. Res., 25, 2024. 6, 7, iii

    work page 2024

  5. [5]

    Fung, and Steven Hoi

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N. Fung, and Steven Hoi. InstructBLIP: Towards general- purpose vision-language models with instruction tuning. NeurIPS, 2024. 1, 2, 6

    work page 2024

  6. [6]

    EV A: Exploring the limits of masked visual represen- tation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EV A: Exploring the limits of masked visual represen- tation learning at scale. In CVPR, 2023. 6

    work page 2023

  7. [7]

    Deep Ensembles: A loss landscape perspective

    Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A loss landscape perspective. arXiv:1912.02757 [stat.ML], 2019. 2

    arXiv 1912

  8. [8]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, pages 1050–1059, 2016. 2

    work page 2016

Show all 49 references
  1. [9]

    ICAN: Instance-centric attention network for human-object interac- tion detection

    Chen Gao, Yuliang Zou, and Jia-Bin Huang. ICAN: Instance-centric attention network for human-object interac- tion detection. In BMVC, 2018. 2

    2018

  2. [10]

    To- wards open-vocabulary scene graph generation with prompt- based finetuning

    Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. To- wards open-vocabulary scene graph generation with prompt- based finetuning. In ECCV, 2022. 1, 2

    2022

  3. [11]

    Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratud- din, and Hamid Laga

    Md. Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratud- din, and Hamid Laga. A comprehensive survey of deep learn- ing for image captioning. ACM Computing Surveys, 51(6): 1–36, 2019. 1

    2019

  4. [12]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700–6709, 2019. 1, 6, i

    2019

  5. [13]

    A. S. M. Iftekhar, Satish Kumar, R. Austin McEver, Suya You, and B. S. Manjunath. GTNet: Guided transformer net- work for detecting human-object interactions. In Pattern Recognition and Tracking XXXIV , pages 192–205. SPIE,

  6. [14]

    Shamma, et al

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32...

    2017

  7. [15]

    Active instruction tuning: Improving cross- task generalization by training on prompt sensitive tasks

    Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. Active instruction tuning: Improving cross- task generalization by training on prompt sensitive tasks. arXiv:2311.00288 [cs.CL], 2023. 2

    2023 arXiv

  8. [16]

    The Open Images Dataset V4: Uni- fied image classification, object detection, and visual rela- tionship detection at scale

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui- jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The Open Images Dataset V4: Uni- fied image classification, object detection, and ...

    1956

  9. [17]

    LISA: Reasoning segmenta- tion via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: Reasoning segmenta- tion via large language model. In CVPR, 2024. 8

    2024

  10. [18]

    Distance weighted cosine similar- ity measure for text classification

    Baoli Li and Liping Han. Distance weighted cosine similar- ity measure for text classification. In Intelligent Data Engi- neering and Automated Learning, 2013. 2

    2013

  11. [19]

    Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. LA VIS: A library for language-vision intelligence. arXiv:2209.09019 [cs.CV] ,

    arXiv

  12. [20]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML,

  13. [21]

    The devil is in the labels: Noisy label correction for robust scene graph generation

    Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In CVPR, pages 18869–18878, 2022. 2

    2022

  14. [22]

    Zero-shot visual relation detection via com- posite visual cues from large language models

    Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot visual relation detection via com- posite visual cues from large language models. NeurIPS,

  15. [23]

    Natural language guided visual relationship de- tection

    Wentong Liao, Bodo Rosenhahn, Ling Shuai, and Michael Ying Yang. Natural language guided visual relationship de- tection. In CVPRW, 2019. 2

    2019

  16. [24]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 1 9

    2023

  17. [25]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in nat- ural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hi- roaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in nat- ural language processing. ACM Computing Surveys, 2023. 2

    2023

  18. [26]

    Visual relationship detection with language priors

    Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. In ECCV, pages 852–869, 2016. 2

    2016

  19. [27]

    How to measure uncertainty in uncertainty sampling for active learning

    Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke H¨ullermeier. How to measure uncertainty in uncertainty sampling for active learning. Machine Learning, 111(1):89– 122, 2022. 2, 4

    2022

  20. [28]

    A literature survey of active machine learn- ing in the context of natural language processing.SICS Tech- nical Report, 2009

    Fredrik Olsson. A literature survey of active machine learn- ing in the context of natural language processing.SICS Tech- nical Report, 2009. 2

    2009

  21. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 1

    2021

  22. [30]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21,

  23. [31]

    Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts

    Shafin Rahman, Salman Khan, and Fatih Porikli. Zero-shot object detection: Learning to simultaneously recognize and localize novel concepts. In ACCV, pages 547–563, 2018. 2

    2018

  24. [32]

    DeiSAM: Segment anything with deictic prompting

    Hikaru Shindo, Manuel Brack, Gopika Sudhakaran, Deven- dra Singh Dhami, Patrick Schramowski, and Kristian Kerst- ing. DeiSAM: Segment anything with deictic prompting. In NeurIPS, 2024. 1, 8, iv

    2024

  25. [33]

    Aditya Siddhant and Zachary C. Lipton. Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study. arXiv:1808.05697 [cs.CL] ,

    arXiv

  26. [34]

    Vision relation transformer for unbi- ased scene graph generation

    Gopika Sudhakaran, Devendra Singh Dhami, Kristian Kerst- ing, and Stefan Roth. Vision relation transformer for unbi- ased scene graph generation. In ICCV, 2023. 2, 6, 7, i

    2023

  27. [35]

    Learning to compose dynamic tree structures for visual contexts

    Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, pages 6619–6628, 2019. 1

    2019

  28. [36]

    Unbiased scene graph generation from bi- ased training

    Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from bi- ased training. In CVPR, pages 3716–3725, 2020. 2, 7, i

    2020

  29. [37]

    LLaMA: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv:2302.13971 [cs.CL], 2023. 6

    2023 arXiv

  30. [38]

    Oytun Ulutan, A. S. M. Iftekhar, and Bangalore S. Manju- nath. VSGNet: Spatial attention network for detecting hu- man object interactions using graph convolutions. In CVPR, pages 13617–13626, 2020. 2

    2020

  31. [39]

    Dai, and Quoc V

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learn- ers. arXiv:2109.01652 [cs.CL], 2021. 2

    2021 arXiv

  32. [40]

    Detectron2

    Yuxin Wu et al. Detectron2. https://github.com/ facebookresearch/detectron2, 2019. 8

    2019

  33. [41]

    Kankanhalli

    Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mo- han S. Kankanhalli. Learning to detect human-object inter- actions with knowledge. In CVPR, 2019. 2

    2019

  34. [42]

    Hauptman

    Caixia Yan, Qinghua Zheng, Xiaojun Chang, Minnan Luo, Chung-Hsing Yeh, and Alexander G. Hauptman. Semantics- preserving graph propagation for zero-shot object detection. IEEE Trans. Image Process., 29:8163–8176, 2020. 2

    2020

  35. [43]

    Semantics-guided contrastive network for zero-shot object detection

    Caixia Yan, Xiaojun Chang, Minnan Luo, Huan Liu, Xiaoqin Zhang, and Qinghua Zheng. Semantics-guided contrastive network for zero-shot object detection. IEEE Trans. Pattern Anal. Mach. Intell., 46(3):1530–1544, 2024. 2

    2024

  36. [44]

    Neural Motifs: Scene graph parsing with global con- text

    Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural Motifs: Scene graph parsing with global con- text. In CVPR, pages 5831–5840, 2018. 1, 2, 3, 6, 7, i

    2018

  37. [45]

    Visual translation embedding network for visual relation detection

    Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat- Seng Chua. Visual translation embedding network for visual relation detection. In CVPR, pages 5532–5540, 2017. 6, 7, i

    2017

  38. [46]

    Youcai et al. Zhang. Recognize anything: A strong image tagging model. In CVPR, pages 1724–1732, 2024. 8

    2024

  39. [47]

    A survey of active learning for natural language processing

    Zhisong Zhang, Emma Strubell, and Eduard Hovy. A survey of active learning for natural language processing. arXiv:2210.10109 [cs.CL], 2022. 2

    2022 arXiv

  40. [48]

    Prototype-based embedding network for scene graph generation

    Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-based embedding network for scene graph generation. In CVPR, 2023. 2, 6, 7, i

    2023

  41. [49]

    bag on table

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gon- zalez, and Ion Stoica. Judging LLM-as-a-judge with MT- Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL], 2023. 6, 7, ...

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.