Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models

Long Hoang Dang; Ngan Le; Quang-Hung Le; Thao Minh Le; Truyen Tran

REVIEW 3 major objections 5 minor 42 references

Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a vision-language model on nested multi-granular expression pairs and decoding from simple to complex lets a 1.6B model beat larger grounded LVLMs on compositional reasoning, with gains up to 10 points.

desk verdict PromViL is a solid, useful contribution to grounded LVLM reasoning: new nested dataset, progressive decoding, and credible zero-shot gains; the main soft spots are label noise from the parser heuristic and ablations that don't isolate inference-time prompting. read the letter →

arxiv 2412.08125 v2 pith:FWZSGL5Z submitted 2024-12-11 cs.CV cs.CLcs.LG

Quang-Hung Le , Long Hoang Dang , Ngan Le , Truyen Tran , Thao Minh Le This is my paper · ORCID

classification cs.CVcs.CLcs.LG

keywords compositionalvisualreasoninggroundinglargevision-languagemodelsmulti-granularalignmentprogressivereferringexpressionGenomegroundedquestionanswering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large vision-language models can be taught grounded compositional reasoning by training them on nested multi-granular vision-language pairs and then prompting them to decode from simple to complex expressions. It introduces PromViL, a framework consisting of the CompoVL dataset (nested expression-box pairs built from Visual Genome) and a progressive decoding scheme where each level's grounded output becomes the cue for the next. The reported results show a 1.6B model fine-tuned on 60K samples with only 4.9% trainable parameters outperforming larger grounded LVLMs: about 9 points on the new CompoVL-hard benchmark, up to 5.5 points on zero-shot referring expression grounding, and roughly 5 and 10 points on accuracy and validity for zero-shot compositional VQA. If these results hold, the path to compositional reasoning in multimodal models may lie in explicit hierarchical grounding rather than ever-larger scale.

What carries the argument

The load-bearing mechanism is a chain of nested multi-granular alignments. CompoVL is built by chaining Visual Genome's ⟨subject, relation, object⟩ predicates that share entities, using an LLM to render the combined predicates as a complex expression and a dependency parser to assign the bounding box of the main head noun to the whole expression; the result is a list of nested expressions at multiple complexity levels. At inference, a constituency parser extracts nested noun-phrase subsequences of the input expression, a dependency parser prunes non-referential level-one items, and the model is prompted level by level with the previous grounded response as context, using next-token prediction trained across all levels. The core objects are the nested expression lists and the progressive prompt chain, which carry the argument: ablation shows that removing intermediate levels or using only one granularity degrades accuracy.

What would settle it

Build a test set of compositional expressions whose referent is not a single head noun (coordinate noun phrases, plural sets, or expressions where the dependency parser's main entity is not the denoted region) and compare PromViL's grounding accuracy against human judgment. A sharp drop would show that the parser's head-noun box is the load-bearing crutch rather than progressive alignment.

Watch

Extended reading notes

Core claim

The central discovery is that aligning language and vision at multiple nested granularities, and using simpler grounded expressions as stepping stones for more complex ones, lets a grounded LVLM reason compositionally where holistic or object-only alignments fail. The authors claim PromViL, fine-tuned on the CompoVL dataset with 4.9% tunable parameters and 60K samples, raises its base model Kosmos-2's accuracy on CompoVL-hard from 59.12 (fine-tuned on the same data without nesting) to 64.07, and on zero-shot RefCOCOg from 60.57 to 64.44, while also improving GQA accuracy and validity. The paper attributes these gains to the chain of progressive multi-granular alignments, supported by ablations that remove intermediate levels.

Load-bearing premise

The dataset and the benchmark both assume that a dependency parser's chosen main noun is what a complex expression refers to, so any expression about a set, or any parser mistake, is paired with the wrong or ambiguous region.

Editorial extensions

If this is right

With the same 1.6B backbone, PromViL outperforms Kosmos-2 by about 4 points on RefCOCOg and about 5.5 points on RefCOCO in zero-shot grounding, and it beats larger zero-shot models like CoVLM 2.8B and Pink 7B.
The roughly 9-point gain on CompoVL-hard and the about 4-point gain on GQA over Kosmos-2 fine-tuned on the same data without nested structure indicate that the nested multi-granular structure, not extra VQA data, drives the improvement.
Ablations show every complexity level is needed: removing intermediate levels drops CompoVL-hard accuracy from 64.07 to 61.22, and keeping only the simplest or only the highest level yields 53.26 or 59.12 respectively, so neither fine-grained nor coarse-grained alignment alone is sufficient.
The performance gap over baselines widens with expression complexity and scene object count, meaning the method specifically targets harder compositional cases rather than providing a uniform boost.
CompoVL-hard serves as a diagnostic benchmark where current grounded LVLMs drop from roughly 84% to 62% accuracy, exposing the compositional grounding failure the paper addresses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same nested-data recipe could transfer to other grounded LVLMs without the parser's head-noun bias; a direct test would be to train PromViL on CompoVL with human-annotated expression-level boxes and compare the gain, isolating how much of the improvement is genuine compositional learning rather than learning the parser's heuristic.
Because the inference chain depends on constituency and dependency parses of the query, the method's accuracy on out-of-distribution syntax (coordinate subjects, non-canonical word order, or set-denoting phrases) would probably drop; evaluating on such a probe set would reveal the parse-sensitivity of progressive decoding.
The reported gains are measured on benchmarks that share Visual Genome's predicate vocabulary and box style, so a testable extension is to measure zero-shot grounding on datasets with different annotation conventions, such as one box per entity versus one box per entire phrase, to see whether the hierarchy transfers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know: this paper delivers a genuinely new mechanism—nested multi-granular V-L pairs from Visual Genome, plus a progressive decoding scheme that feeds lower-level grounding answers into higher-level prompts. The zero-shot results on RefCOCO*, GQA, and Visual7W are the strongest evidence; they're external benchmarks not built from the generation pipeline, so the central claim isn't circular in the obvious way. The paper is honest about its setup and the code/data release is a plus.

What's new: the CompoVL dataset (60K instances, nested expressions up to level 3+) and the progressive prompting algorithm. The ablations in Table 5 show that removing intermediate levels hurts, which supports the multi-granular training idea. The gains over Kosmos-2 and MiniGPTv2 baselines are consistent across tasks.

Where it's soft: First, the stress-test about the dependency parser is real. Every bounding box for a complex expression is inherited from the main entity identified by the parser over LLM-generated text. If the parser picks the wrong head noun, the label is wrong. The human evaluation (92.48% bbox accuracy on 2% of CompoVL-hard) is reassuring but doesn't compare against the VG predicate's intended subject; it judges the box against the generated caption, so it can't detect systematic parser errors. This is a genuine weakness, though not necessarily fatal—the external zero-shot results suggest the training signal is still useful.

Second, the ablations change training data, not inference decoding. The reader is right: we don't learn whether progressive prompting at inference is what drives the gains, or just the multi-granular training data. A proper ablation would run the same model with and without the progressive chain at inference.

Third, the supervised RefCOCO* comparison may be contaminated by image overlap between Visual Genome and MS COCO. The paper doesn't address this directly, so those numbers should be re-evaluated on non-overlapping splits. Minor: no error bars, and the claim of "unbounded complexity" is overstated; the data only goes to a few levels.

Overall: a worthwhile paper with a load-bearing but not fatal assumption. The parser-labeling issue deserves scrutiny, but the external zero-shot evidence carries the core idea. I'd send it to peer review with a request for the overlap check, an inference-only ablation, and error bars. It's a solid contribution for anyone working on grounded LVLM reasoning; I'd cite the dataset and method if I were in that space.

Referee Report

3 major / 5 minor

Summary. The paper proposes PromViL, a method that augments grounded LVLMs with multi-granular, nested vision-language supervision and a progressive decoding procedure. It constructs CompoVL from Visual Genome predicates by chaining related triples through an LLM and assigning each generated expression to the VG bounding box of the dependency-parser-selected head noun. Training uses LoRA fine-tuning of Kosmos-2 (or MiniGPTv2) with a next-token loss over expression levels, conditioning each level on the previous level's expression and box tokens. At inference, a constituency/dependency parser recovers nested subsequences and the model is prompted level by level. The authors report gains over base and fine-tuned baselines on CompoVL-hard (about 9 points over Kosmos-2), zero-shot RefCOCO/RefCOCO+/RefCOCOg grounding, and zero-shot GQA/GQA-OOD/Visual7W, plus ablations showing that all nesting levels matter.

Significance. The central idea—hierarchical visual-language alignment plus progressive decoding—is timely and, if the concerns below are addressed, could yield a lightweight recipe (4.9% LoRA parameters, 60K fine-tuning samples, single V100). The paper includes useful controls: F-Kosmos-2 uses the same CompoVL data without nested subsequences; Kosmos-8K/16K control for the amount of VQA data; and the zero-shot RefCOCO*, GQA, and Visual7W benchmarks are not generated by the CompoVL pipeline. The Table 5 ablation clearly shows that intermediate nesting levels contribute beyond merely having the highest-level expression. These strengths make the paper potentially publishable, but the load-bearing data-labeling heuristic and several missing controls require additional work before the claims are fully established.

major comments (3)

[Methods — Multi-granular Compositional V-L Dataset / Annotation's reliability] The sentence "The visual bounding boxes are then chosen as bounding boxes of the identified main entities in VG" is load-bearing: every CompoVL training target and every CompoVL-hard evaluation label inherits the dependency parser's head-noun choice. The manuscript does not validate that this choice agrees with the intended subject of the VG predicate chain, and the 2% human evaluation asks annotators whether the box is correct for the generated caption, not whether the parser selected the VG predicate's subject. It also reports only aggregate naturalness/uniqueness/bbox rates, with no breakdown by complexity level. Because the same parser is reused at inference to prune level-one expressions, parser errors can transfer into the prompting chain. Since CompoVL-hard is produced by the same pipeline that creates the training labels, the 8.7-point CompoVL-hard gain is not by itself evidence of compositional grounding; the external zero-shot results carry that claim. I ask for a direct parser validation (head-noun agreement with VG subject annotations, bbox agreement per complexity level, and ambiguity rates on higher-complexity items) and for an analysis of whether the widening accuracy gap in Fig. 5 survives after conditioning on label quality.
[Experiments — Referring expression tasks (Table 2)] The supervised RefCOCO/RefCOCO+/RefCOCOg results are potentially inflated by image overlap between Visual Genome and MS COCO. RefCOCO* test images come from COCO, and CompoVL is derived from VG, which shares many images with COCO; the fine-tuned PromViL‡ has therefore plausibly seen the test images during CompoVL training. Even if the Pink and MiniGPTv2 baselines were also trained on VG region-caption data, the comparison does not separate generalization from memorization. Please quantify the overlap between the CompoVL training images and the RefCOCO* test splits, and report the supervised comparison on the non-overlapping subset (or otherwise control for overlap).
[Methods — Training and Inference (Eq. 1, Algorithm 1); Model Analysis — Table 5] The experimental design does not isolate the inference-time progressive mechanism from the benefit of being trained on nested data. Table 5 varies the training data (all levels, intermediate removed, highest only, simplest only), but there is no condition that trains on all levels and then decodes flat, or trains without nesting and then decodes progressively. Moreover, Eq. (1) teacher-forces the previous level's ground-truth box at training time, while Algorithm 1 feeds generated boxes at inference; no analysis reports the accuracy of the intermediate generated boxes or how errors propagate. Please add the missing cross-condition (e.g., full training with single-prompt decoding, or a parser-based decomposition ablated) and report intermediate-box accuracy. It should also be stated explicitly whether F-Kosmos-2 in Table 1 uses flat or progressive decoding, since this determines what that comparison controls for.

minor comments (5)

[General] Several references to 'Sec.' and 'Appendix' are unresolved; for example, the text says 'See Sec. ' after presenting CompoVL-hard, and the prompts and evaluation interface are said to be in the Appendix but do not appear in this version. Please include the appendix or point to the repository.
[General] There are typographical errors: 'dependancies' in the Inference paragraph, 'thile@@uark.edu' in the author list, 'spatial tokensyi' in the Training paragraph, and 'To quantify the the relations' in Model Analysis and Ablation Studies.
[Methods — Algorithm 1] Algorithm 1 line 8 contains 'yc ← yi', which appears to be a typo for returning the final generated sequence; please correct.
[Experiments — Table 2] The footnote '(*) are supervised on region-caption data from VG' is ambiguous: it is unclear which rows the asterisk applies to and whether PromViL‡ was additionally trained on the RefCOCO* training splits. Please state the exact training data for each supervised row.
[Abstract] The abstract's 'nearly 5 point and 10 point increases in accuracy and validity' should be calibrated to Table 3: the accuracy gap is 45.07 versus 40.55 (4.52 points) and the validity gap is 83.47 versus 74.82 (8.65 points).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central grounding gains are confirmed on independent human-annotated benchmarks, while CompoVL-hard is an in-house benchmark that is human-validated but not used as a derivation.

full rationale

The paper's central claim is that hierarchical multi-granular V-L training plus progressive decoding improves grounded compositional reasoning. That claim is not circular: the method is evaluated on RefCOCO/RefCOCO+/RefCOCOg, GQA, GQA-OOD, and Visual7W, whose labels are human-annotated and independent of the CompoVL construction pipeline. The gains on those external benchmarks provide independent evidence that the approach transfers beyond the generated dataset. The only in-house evaluation is CompoVL-hard, whose labels are created by the same pipeline (VG predicates, LLM-generated text, dependency-parser main-entity boxes) used for training; this makes CompoVL-hard an in-domain benchmark, but the paper does not derive any prediction from the pipeline, and it reports a 92.48% human-validated bounding-box accuracy on a 2% sample. There is no fitted parameter later renamed as a prediction, no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled in through citation. The dependency-parser head-noun heuristic is a data-validity caveat for CompoVL-hard, not a circular step: at inference the model must predict the bounding box from the image rather than receiving the parser's chosen box, and the external benchmarks do not share the generation rule. Overall, the derivation chain is self-contained and the main empirical claims are supported by independent evaluations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters are fitted to benchmarks; the method relies instead on modeling assumptions about parser correctness and dataset completeness. CompoVL is a new dataset artifact, not a conceptual invented entity in the sense of a new physical entity or mechanism. The main load-bearing assumptions are the parser-based target assignment and the match between parse-time and training-time decomposition structure.

assumptions (4)

domain assumption The head noun selected by the dependency parser identifies the visual target of a complex referring expression.
Used to generate CompoVL labels and to prune inference decompositions. If the parser selects the wrong head, the expression is paired with the wrong bounding box. Location: Methods, 'Multi-granular Compositional V-L Dataset'.
domain assumption Visual Genome object annotations and predicate triples are complete enough to cover all entities mentioned in LLM-generated descriptions.
The data generator is instructed not to invent objects, but if it nevertheless produces text mentioning an entity absent from VG annotations, no correct bounding box exists for that entity. Location: Methods, 'Multi-granular Compositional V-L Dataset'.
domain assumption Constituency parsing at inference yields nested noun-phrase subsequences that mirror CompoVL's construction.
The progressive decoder relies on the parser to recover the same nesting structure used during training. Parse errors propagate through the whole prompting chain. Location: Methods, 'Training and Inference' and Algorithm 1.
domain assumption The LLM used for data generation strictly follows the provided VG predicates without inventing new objects or relations.
Prompt-level mitigation is described, but the paper provides no formal guarantee; any hallucinated entity would produce a noisy or ungroundable training sample. Location: Methods, data generation paragraph.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models." pith.science (2026). https://pith.science/paper/FWZSGL5Z

@misc{pith2026241208125,
  author       = {Pith},
  title        = {Pith review of: Progressive Multi-granular Alignments for Grounded Reasoning in Large Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FWZSGL5Z}},
  note         = {Machine review of arXiv:2412.08125}
}

read the original abstract

Existing Large Vision-Language Models (LVLMs) excel at matching concepts across multi-modal inputs but struggle with compositional concepts and high-level relationships between entities. This paper introduces Progressive multi-granular Vision-Language alignments (PromViL), a novel framework to enhance LVLMs' ability in performing grounded compositional visual reasoning tasks. Our approach constructs a hierarchical structure of multi-modal alignments, ranging from simple to complex concepts. By progressively aligning textual descriptions with corresponding visual regions, our model learns to leverage contextual information from lower levels to inform higher-level reasoning. To facilitate this learning process, we introduce a data generation process that creates a novel dataset derived from Visual Genome, providing a wide range of nested compositional vision-language pairs. Experimental results demonstrate that our PromViL framework significantly outperforms baselines on various visual grounding and compositional question answering tasks. The code is available at: https://github.com/lqh52/PromViL.

Figures

Figures reproduced from arXiv: 2412.08125 by the authors.

**Figure 1.** Comparison with Existing LVLMs: (a) Coarsegrained: Whole image/region with full text, lacks object details. (b) Fine-grained: Simple phrases and bounding boxes, lacks relational context. (c) PromViL employs hierarchical multi-granular associations, progressively utilizing simpler concepts as cues to understand more complex ones. grounding of textual information (Li et al. 2023a) and subsequently to suboptimal per… view at source ↗

**Figure 2.** Overview of our PromViL framework. (a) Training: Learn multi-level visual entities-textual expression associations. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Multi-granular Compositional V-L Data Genera [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Qualitative results on CompoVL-hard. Solid [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Accuracy comparison between PromViL and MiniGPTv2 on the CompoVL-hard dataset. Experiments Implementation Details: Unless otherwise stated, our PromViL is achieved by finetuning Kosmos-2 (Peng et al. 2024) on the CompoVL dataset, adhering to its default hyperparameter…

**Figure 6.** Figure 6: Accuracy comparison between PromViL and Kosmos-8K on the GQA val dataset. Referring expression tasks: To validate our proposed method PromViL’s generalization capability on visual grounding task, we conduct evaluations on well-established benchmarks: RefCOCOg (Mao et a…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 30 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems

work page 2022
[4]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023
[5]

Bottou, L. 2014. From machine learning to machine reasoning: An essay. Machine learning

work page 2014
[6]

Chen, G.; Shen, L.; Shao, R.; Deng, X.; and Nie, L. 2024. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

work page 2024
[7]

Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

arXiv 2023
[8]

Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement

work page 1960

Show all 42 references

[9]

Cresswell, M. J. 2016. Logics and languages. Routledge

2016
[10]

H.; Le, T

Dang, L. H.; Le, T. M.; Le, V.; and Tran, T. 2021. Hierarchical object-oriented spatio-temporal reasoning for video question answering. arXiv preprint arXiv:2106.13432

2021 arXiv
[11]

Gao, Y.; Liu, J.; Xu, Z.; Zhang, J.; Li, K.; Ji, R.; and Shen, C. 2022. Pyramidclip: Hierarchical feature alignment for vision-language model pretraining. Advances in neural information processing systems

2022
[12]

Hallgren, K. A. 2012. Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in quantitative methods for psychology

2012
[13]

Honnibal, M.; Montani, I.; Van Landeghem, S.; and Boyd, A. 2020. spaCy: Industrial-strength Natural Language Processing in Python

2020
[14]

J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[15]

K.; Patra, B.; et al

Huang, S.; Dong, L.; Wang, W.; Hao, Y.; Singhal, S.; Ma, S.; Lv, T.; Cui, L.; Mohammed, O. K.; Patra, B.; et al. 2024. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36

2024
[16]

A.; and Manning, C

Hudson, D. A.; and Manning, C. D. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

2019
[17]

Hupkes, D.; Dankers, V.; Mul, M.; and Bruni, E. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research

2020
[18]

M.; and Partee, B

Janssen, T. M.; and Partee, B. H. 1997. Compositionality. In Handbook of logic and language. Elsevier

1997
[19]

Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D

Jiang, A. Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Hanna, E. B.; Bressand, F.; et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

2024 arXiv
[20]

Kazemzadeh, S.; Ordonez, V.; Matten, M.; and Berg, T. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)

2014
[21]

Kervadec, C.; Antipov, G.; Baccouche, M.; and Wolf, C. 2021. Roses are red, violets are blue... but should vqa expect them to? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2021
[22]

Kitaev, N.; Cao, S.; and Klein, D. 2019. Multilingual Constituency Parsing with Self-Attention and Pre-Training. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

2019
[23]

A.; et al

Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision

2017
[24]

M.; Le, V.; Venkatesh, S.; and Tran, T

Le, T. M.; Le, V.; Venkatesh, S.; and Tran, T. 2020. Hierarchical conditional relation networks for video question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

2020
[25]

Li, J.; Chen, D.; Hong, Y.; Chen, Z.; Chen, P.; Shen, Y.; and Gan, C. 2023 a . Covlm: Composing visual entities and relationships in large language models via communicative decoding. arXiv preprint arXiv:2311.03354

2023 arXiv
[26]

Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023 b . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. International conference on machine learning

2023
[27]

Liu, F.; Emerson, G.; and Collier, N. 2023. Visual spatial reasoning. In Transactions of the Association for Computational Linguistics

2023
[28]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2024. Visual instruction tuning. Advances in neural information processing systems

2024
[29]

Liu, S.; Zeng, Z.; Ren, T.; Li, F.; Zhang, H.; Yang, J.; Li, C.; Yang, J.; Su, H.; Zhu, J.; et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499

2023 arXiv
[30]

O.; Gandhi, M.; Gao, I.; and Krishna, R

Ma, Z.; Hong, J.; Gul, M. O.; Gandhi, M.; Gao, I.; and Krishna, R. 2023. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2023
[31]

L.; and Murphy, K

Mao, J.; Huang, J.; Toshev, A.; Camburu, O.; Yuille, A. L.; and Murphy, K. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition

2016
[32]

Nivre, J. 2008. Algorithms for Deterministic Incremental Dependency Parsing. Computational Linguistics

2008
[33]

Peng, Z.; Wang, W.; Dong, L.; Hao, Y.; Huang, S.; Ma, S.; Ye, Q.; and Wei, F. 2024. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations

2024
[34]

A.; Wang, L.; Cervantes, C

Plummer, B. A.; Wang, L.; Cervantes, C. M.; Caicedo, J. C.; Hockenmaier, J.; and Lazebnik, S. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision

2015
[35]

Shao, S.; Li, Z.; Zhang, T.; Peng, C.; Yu, G.; Zhang, X.; Li, J.; and Sun, J. 2019. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision

2019
[36]

Wang, P.; Yang, A.; Men, R.; Lin, J.; Bai, S.; Li, Z.; Ma, J.; Zhou, C.; Zhou, J.; and Yang, H. 2022. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International conference on machine learning

2022
[37]

Xuan, S.; Guo, Q.; Yang, M.; and Zhang, S. 2024. Pink: Unveiling the power of referential comprehension for multi-modal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

2024
[38]

C.; and Berg, T

Yu, L.; Poirson, P.; Yang, S.; Berg, A. C.; and Berg, T. L. 2016. Modeling context in referring expressions. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14

2016
[39]

Yuksekgonul, M.; Bianchi, F.; Kalluri, P.; Jurafsky, D.; and Zou, J. 2023. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations

2023
[40]

Zeng, Y.; Zhang, X.; and Li, H. 2021. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276

2021 arXiv
[41]

Zhao, T.; Zhang, T.; Zhu, M.; Shen, H.; Lee, K.; Lu, X.; and Yin, J. 2022. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221

2022 arXiv
[42]

Zhu, Y.; Groth, O.; Bernstein, M.; and Fei-Fei, L. 2016. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition

2016

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems

work page 2022

[4] [4]

Bai, J.; Bai, S.; Yang, S.; Wang, S.; Tan, S.; Wang, P.; Lin, J.; Zhou, C.; and Zhou, J. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023

[5] [5]

Bottou, L. 2014. From machine learning to machine reasoning: An essay. Machine learning

work page 2014

[6] [6]

Chen, G.; Shen, L.; Shao, R.; Deng, X.; and Nie, L. 2024. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

work page 2024

[7] [7]

Chen, J.; Zhu, D.; Shen, X.; Li, X.; Liu, Z.; Zhang, P.; Krishnamoorthi, R.; Chandra, V.; Xiong, Y.; and Elhoseiny, M. 2023. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478

arXiv 2023

[8] [8]

Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement

work page 1960