Pith. sign in

REVIEW 4 major objections 5 minor 87 references

Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieving visually similar in-context images inflates image-captioning scores mainly because the model copies the demonstration captions, not because it understands the query image better.

desk verdict A solid empirical sweep with reusable attention metrics, but the shortcut-copying headline needs a baseline before it can be trusted. read the letter →

arxiv 2507.08021 v1 pith:F4TPJ2CO submitted 2025-07-08 cs.CL cs.AI

Li Li , Yongliang Wu , Jingze Zhu , Jiawei Peng , Jianfei Cai , Xu Yang This is my paper
classification cs.CLcs.AI
keywords in-contextlearninglargemultimodalmodelsimagecaptioningdemonstrationconfigurationattentionanalysishallucinationshortcutCIDEr
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies how the configuration of in-context examples (ICEs) affects image captioning in large multimodal models, and it tries to explain why some configurations help or hurt. It claims that increasing the number of shots improves linguistic coherence but can degrade visual-text alignment and increase hallucination, while retrieving visually similar images raises CIDEr scores largely through a copying mechanism the authors call short-cut inference. The paper also argues that the quality and linguistic style of ICE captions strongly steer model output, and that two models with nearly identical architectures behave differently because of pre-training corpus differences. Internally, it identifies three attention patterns and proposes three attention-based metrics to quantify them. The work matters because it suggests previously reported gains from similarity-based retrieval in multimodal in-context learning may be partly an artifact of copying rather than evidence of stronger visual understanding.

What carries the argument

The load-bearing objects are three attention-based ratios. ACAR (Anchor-to-Context Attention Ratio) measures how much query tokens attend to anchor tokens such as the initial token, image token, period marks, and chunk delimiters relative to other context tokens, quantifying the anchor aggregation pattern. IEAR (Intra-Extra Attention Ratio) measures the average attention within one in-context example relative to attention across different in-context examples, quantifying the emergent attention window. VCAR (Visual-to-Caption Attention Ratio) indirectly estimates how much of the query-to-query attention is attributable to the query image by comparing self-attention with and without the image, thereby separating visual reliance from reliance on in-context caption text. These metrics are what connect the external configuration findings to internal behavior, and the Flamingo gated cross-attention mechanism, in which captions act as queries and image features as keys and values, is cited as the architectural reason that visually similar in-context images amplify the influence of their captions.

What would settle it

Swap the vision encoders and language backbones between OpenFlamingo and IDEFICS, or run each model at the other's sampling temperature, and rerun the 32-shot captioning evaluation: if the hallucination gap and VCAR gap follow the components rather than the pre-training corpus, the corpus-bias explanation fails. A second check: under similarity-based retrieval, replace ICE captions with semantically plausible but image-mismatched captions; if CIDEr stays inflated while CLIPScore drops, copying is confirmed as the driver of the CIDEr gain.

Watch

Extended reading notes

Core claim

The central claim is that in multimodal in-context learning for image captioning, the standard heuristic of retrieving in-context images most similar to the query does increase CIDEr, but this increase is driven by the model copying in-context captions rather than by improved visual understanding. The paper supports this by showing that similarity-based retrieval raises short-cut CIDEr, a newly defined measure of caption copying, by orders of magnitude while simultaneously lowering VCAR, an attention-based estimate of how much the model relies on the query image; CLIPScore does not improve and hallucination metrics worsen. A second major claim is that adding shots mainly helps linguistic pattern recognition, not multimodal reasoning, so CIDEr gains can coexist with worse visual-text alignment and more hallucination in one of the two models. The paper further finds that ICE caption quality has a two-sided effect that image similarity amplifies: high-quality captions help and low-quality captions hurt, with similar images intensifying either influence through the cross-attention mechanism. It also identifies anchor tokens and emergent attention windows as characteristic attention patterns, and shows that keeping only anchor tokens in middle layers can prune over 70 percent of the sequence while preserving most captioning performance.

Load-bearing premise

The causal story that attributes the behavioral differences between OpenFlamingo and IDEFICS to pre-training corpus characteristics treats the models' remaining differences—vision encoder, language backbone, and sampling temperature—as negligible, yet no controlled ablation isolates the corpus as the sole cause.

Editorial extensions

If this is right

  • Similarity-based retrieval should not be assumed to improve multimodal in-context learning, because its CIDEr gains can mask degraded visual-text alignment and higher hallucination.
  • Higher-shot settings need evaluation metrics that capture visual grounding, since CIDEr improvements alone do not imply better multimodal reasoning.
  • Attention-based metrics such as VCAR could serve as diagnostics for detecting hallucination-prone or shortcut-prone configurations before deployment.
  • Anchor-centric pruning demonstrates that most context tokens in middle layers can be dropped with small performance loss, a result that becomes more valuable as LMM context windows grow toward many-shot regimes.
  • Pre-training corpus sequence length and image-text quality emerge as plausible levers for improving LMM in-context robustness, offering guidance for future data curation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shortcut effect generalizes beyond the two tested models, then earlier benchmarks reporting similarity-based retrieval gains in multimodal few-shot settings may have overstated genuine visual understanding; re-evaluating those settings with short-cut CIDEr or CLIPScore would expose how much of the gain is copying.
  • The emergent attention window suggests each in-context example is processed nearly independently in middle layers, which implies a testable prediction: shuffling the order of in-context examples should have only a small effect on generation in these models, a consequence the paper does not directly test.
  • The pruning result could be extended to extreme many-shot regimes: if future LMMs handle contexts of thousands of tokens, anchor-centric compression may yield much larger speedups than the roughly 1K-token settings used here.
  • VCAR, despite its acknowledged non-strict correlation with hallucination, could be developed into a real-time monitoring signal during decoding, though the paper's own caveat indicates it would need to be combined with other predictors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper investigates multimodal in-context learning for image captioning on OpenFlamingo v2-9B and IDEFICS v1-9B. Externally, it varies three configuration dimensions—shot number, image retrieval (Random Sampling vs. similarity-based image-to-image retrieval, SIIR), and caption assignment (human, machine-generated, and model-guided human captions)—and evaluates outputs with CIDEr, CLIPScore, CHAIR, and a newly proposed ShortCut-CIDEr metric. Internally, it identifies three attention patterns (anchor tokens, emergent attention windows, and shortcut inference), proposes attention-based metrics ACAR, IEAR, and VCAR, and conducts preliminary acceleration experiments via selective masking and pruning. The paper reports six findings, the most consequential being that similarity-based retrieval inflates CIDEr through copying of in-context captions rather than through improved visual understanding.

Significance. If the central claims hold, the paper would be a useful empirical contribution: Finding 5 challenges the common assumption that similarity-based demonstration retrieval straightforwardly helps multimodal in-context learning, and the attention-based metrics offer a lightweight way to quantify shortcut behavior and hallucination. The paper also deserves credit for covering a large combinatorial space of configurations, using multiple complementary metrics, and shipping concrete quantitative results for attention-driven acceleration. However, the shortcut claim rests on an unvalidated metric, and several causal conclusions are confounded by uncontrolled model differences and by the absence of statistical grounding. The qualitative findings are plausible and the paper is worth revising, but the load-bearing evidence for the headline conclusion needs substantial strengthening.

major comments (4)
  1. [Section IV-B3, Figure 9, Finding 5] ShortCut-CIDEr is computed as CIDEr between the generated caption and the first four ICE captions. Because CIDEr is a soft n-gram overlap score rather than a copy detector, and because SIIR selects images visually similar to the query, the human captions of SIIR-selected ICEs will tend to share n-grams with a correct caption of the query. The paper never reports the control baseline CIDEr between ground-truth query captions and SIIR-selected ICE captions. Without that baseline, the large ShortCut-CIDEr jump under SIIR does not establish copying; it may instead reflect the model correctly describing a visually similar image. Please report this baseline and an exact-match or longest-common-substring copy measure, then re-evaluate Finding 5.
  2. [Section IV-B1, Findings 2 and 6] The causal claim that behavioral differences between OpenFlamingo and IDEFICS "largely stem from the used training datasets" is not identifiable from the current experiments: the two models differ in vision encoder (CLIP ViT vs. OpenCLIP ViT), language backbone (MPT vs. LLaMA), and sampling temperature (0.2 vs. 0.1). No ablation holds these factors fixed. Please add a controlled comparison (e.g., temperature-matched sampling or a same-backbone/different-data setting) or explicitly reframe Findings 2 and 6 as correlational observations that are consistent with, but not proof of, pre-training data effects.
  3. [Section IV-C, Figure 5] The paper reports "dimension-wise averaged results" over all combinations of two image-retrieval methods and eleven caption-assignment strategies, but no error bars, confidence intervals, or significance tests are reported. Claims such as "IDE demonstrates significantly greater gains than OF" and "SIIR introduces more hallucinations" cannot be evaluated without knowing the variance across the averaged configurations. Please report per-configuration spreads or paired statistical tests, and state explicitly how many configurations each averaged point represents.
  4. [Section III-C3, Eqs. (11)-(16), Section IV-B3] VCAR is an indirect attention-difference proxy: it estimates visual attention flow by subtracting self-attention with and without the query image, and the paper itself describes it as a partial proxy that is not strictly negatively correlated with hallucination. Nevertheless, VCAR is used as evidence for the shortcut explanation in Findings 5 and 6. Please validate VCAR against a direct behavioral measure (e.g., verbatim copying rate or human-annotated shortcut cases) and quantify the correlation, or weaken the causal conclusions drawn from VCAR.
minor comments (5)
  1. [References] Several references are duplicated: [3] and [43] are the same Radford et al. paper; [23] is duplicated as [68]; [28] and [32] are the same OBELICS paper; [71] and [73] are the same "Label Words Are Anchors" paper. Please consolidate the bibliography.
  2. [Figure 2] The red arrows and the red dashed triangle in Figure 2 are difficult to see in grayscale or small print; please enlarge the annotations or add textual callouts.
  3. [Table II] The notation "Anchor-Centric Masking(10-30)" and "Anchor-Centric Masking(15-30)" is never explained; the caption or text should state that the numbers denote decoder layer ranges.
  4. [Section IV-B3] The definition of ShortCut-CIDEr says "first four ICE captions" without clarifying whether these are the first four in sequence order or the first four retrieved examples, and whether the order is held fixed across settings; please specify.
  5. [Throughout] The prose contains grammatical errors and awkward phrasings (e.g., "Researches [1], [11]–[14] have shown", "the dataset of OF contains sequences with an average of fewer tokens") and would benefit from a careful copyedit.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the paper's central claims rest on independent external benchmarks (CIDEr, CLIPScore, CHAIR) and a priori attention-based metrics, with only minor non-load-bearing self-citation.

full rationale

The paper's main findings are not forced by construction. The external evaluation uses standard metrics (CIDEr, CLIPScore, CHAIR) that are independent of the paper's proposed strategies; no parameter is fitted to a subset and then used to predict a closely related quantity. The internal metrics (ACAR, IEAR, VCAR) are defined algebraically from attention weights, not calibrated against the conclusions they are used to explain, so they do not make the conclusions true by definition. Finding 5's claim that similarity-based retrieval inflates CIDEr via shortcut copying is supported by ShortCut-CIDEr and VCAR evidence; while a missing ground-truth baseline weakens the causal interpretation, this is a validation gap rather than a circular reduction. The paper cites its own prior works [41] and [65] when introducing the notion of shortcut inference, but these citations are contextual and not load-bearing: the present paper defines and measures the phenomenon with its own metrics. Findings 1 and 2 attribute model differences to pre-training data characteristics through a post hoc explanation; that is speculative causal reasoning, not circularity. Overall, the derivation chain is self-contained, and the minor self-citations do not undermine the independent empirical content.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities. Its new constructs are measurement tools (ACAR, IEAR, VCAR) and token category definitions (anchor, query, context), plus hand-chosen settings for temperature, masking layers, pruning start layers, and caption quality thresholds.

free parameters (4)
  • sampling temperature = OpenFlamingo 0.2, IDEFICS 0.1
    Different temperatures for the two models may influence hallucination and output diversity, yet comparisons treat them as incidental.
  • anchor-centric masking layer range = layers 10-30 and 15-30
    Hand-selected layer ranges for masking tests; changing this range changes the reported quality-speed trade-off.
  • pruning start layer = layers 10, 15, and 20
    Hand-selected starting layers for token pruning in the acceleration experiments.
  • caption quality thresholds = CIDEr 60/80/135 for TF-generated captions
    Hand-defined low/medium/high quality levels used to categorize machine-generated captions.
assumptions (3)
  • domain assumption Attention weights in gated cross-attention layers reflect how much visual information the model actually uses
    The VCAR metric (Eq. 11-16) builds on this premise to attribute short-cuts and hallucinations to attention allocation.
  • domain assumption Performance differences between OpenFlamingo and IDEFICS are primarily due to pre-training dataset differences
    Sections IV-B1 and Finding 2 rely on this assumption despite the models also differing in vision encoder, language backbone, and temperature.
  • standard math CIDEr, CLIPScore, and CHAIR respectively capture linguistic quality, visual-text alignment, and hallucination
    These metrics are standard in image captioning and are used as external benchmarks for the findings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis." pith.science (2026). https://pith.science/paper/F4TPJ2CO

@misc{pith2026250708021,
  author       = {Pith},
  title        = {Pith review of: Unveiling Effective In-Context Configurations for Image Captioning: An External & Internal Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F4TPJ2CO}},
  note         = {Machine review of arXiv:2507.08021}
}
read the original abstract

The evolution of large models has witnessed the emergence of In-Context Learning (ICL) capabilities. In Natural Language Processing (NLP), numerous studies have demonstrated the effectiveness of ICL. Inspired by the success of Large Language Models (LLMs), researchers have developed Large Multimodal Models (LMMs) with ICL capabilities. However, explorations of demonstration configuration for multimodal ICL remain preliminary. Additionally, the controllability of In-Context Examples (ICEs) provides an efficient and cost-effective means to observe and analyze the inference characteristics of LMMs under varying inputs. This paper conducts a comprehensive external and internal investigation of multimodal in-context learning on the image captioning task. Externally, we explore demonstration configuration strategies through three dimensions: shot number, image retrieval, and caption assignment. We employ multiple metrics to systematically and thoroughly evaluate and summarize key findings. Internally, we analyze typical LMM attention characteristics and develop attention-based metrics to quantify model behaviors. We also conduct auxiliary experiments to explore the feasibility of attention-driven model acceleration and compression. We further compare performance variations between LMMs with identical model design and pretraining strategies and explain the differences from the angles of pre-training data features. Our study reveals both how ICEs configuration strategies impact model performance through external experiments and characteristic typical patterns through internal inspection, providing dual perspectives for understanding multimodal ICL in LMMs. Our method of combining external and internal analysis to investigate large models, along with our newly proposed metrics, can be applied to broader research areas.

Figures

Figures reproduced from arXiv: 2507.08021 by the authors.

Figure 1
Figure 1. External Configuration Strategies. (a) The overall pipeline: it first retrieves in-context images from an image corpus, then constructs demonstration sequences by assigning a corresponding caption to each in-context image. (b) Two image￾retrieving methods: Random Sampling (RS) and Similarity-based Image-to-Image Retrieval (SIIR). (c) Four caption-assigning methods: using human-labelled ground-truth annotations, mach… view at source ↗
Figure 2
Figure 2. Two typical internal attention patterns. (a) Anchor Token. (c) Emergent Attention Window. (b) and (d) show a visualization example of these two patterns (4-shot, IDEFICS). For greater clarity, we only show the attention maps of the first two ICEs with the initial token dropped. The red arrows in (b) point to the anchor tokens, which are the image token [<image>], the punctuation marks [.] at the end of each ICE, and… view at source ↗
Figure 3
Figure 3. Attention-based metrics. (a) The input token sequence of LMM is divided into different components. (b) ACAR calculation schematic. (c) IEAR calculation schematic. (d) VCAR calculation schematic. Vision Encoder Image K, V Q <image> A zebra eating grass . Tokenization & Embedding Caption GATED XATTN-DENSE LM block GATED XATTN-DENSE layer [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The gated cross attention of Flamingo. Flamingo integrates visual information into text tokens via gated cross￾attention layers added to the LLM, thus achieving multimodal capabilities. S k I =  (i, j) : i ∈ Ik , j ∈ Ik [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The experimental results from the shot perspective. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The experimental results from assigning caption perspective. The dashed line indicates the score of First Human-Labelled [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: (a) demonstrates the bipolar impact of caption qual [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Three visualization examples. (a) Two captions generated by OF and IDE in 16-shot setting. The caption of OF has a low CIDEr score and contains hallucinations (“bench”), while that of IDE is more accurate. (b) An example of OF under SIIR 32-shot setting. Using three qu…
Figure 9
Figure 9. Figure 9: The experimental results from the image retrieval perspective. The dashed line indicates the score of Random Sampling [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: The average results of ACAR and IEAR for two [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Model pruning. To accelerate inference, non-anchor tokens in ICE are pruned at middle layer K. All tokens are then restored at the prediction layer using the hidden states of pruned layer to maintain performance. a local window that only calculates the self-attention …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

87 extracted references · 32 canonical work pages

  1. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  2. [4]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [5]

    Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Computing Surveys, vol. 55, no. 9, pp. 1–35, 2023

    2023

  4. [6]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382, 2023

    arXiv 2023

  5. [7]

    Selective annota- tion makes language models better few-shot learners,

    H. Su, J. Kasai, C. H. Wu, W. Shi, T. Wang, J. Xin, R. Zhang, M. Ostendorf, L. Zettlemoyer, N. A. Smith et al. , “Selective annota- tion makes language models better few-shot learners,” arXiv preprint arXiv:2209.01975, 2022

    arXiv 2022

  6. [8]

    Making pre-trained language models better few-shot learners,

    T. Gao, A. Fisch, and D. Chen, “Making pre-trained language models better few-shot learners,” arXiv preprint arXiv:2012.15723 , 2020

    arXiv 2012

  7. [9]

    How can we know what language models know?

    Z. Jiang, F. F. Xu, J. Araki, and G. Neubig, “How can we know what language models know?” Transactions of the Association for Computational Linguistics, vol. 8, pp. 423–438, 2020

    2020

  8. [10]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

Show all 87 references
  1. [11]

    Learning to retrieve prompts for in-context learning,

    O. Rubin, J. Herzig, and J. Berant, “Learning to retrieve prompts for in-context learning,” arXiv preprint arXiv:2112.08633 , 2021

    2021 arXiv

  2. [12]

    Active example selection for in-context learning,

    Y . Zhang, S. Feng, and C. Tan, “Active example selection for in-context learning,” arXiv preprint arXiv:2211.04486 , 2022

    2022 arXiv

  3. [14]

    Demystifying prompts in language models via perplexity estimation,

    H. Gonen, S. Iyer, T. Blevins, N. A. Smith, and L. Zettlemoyer, “Demystifying prompts in language models via perplexity estimation,” arXiv preprint arXiv:2212.04037 , 2022

    2022 arXiv

  4. [15]

    What makes good in-context examples for gpt- 3?

    J. Liu, D. Shen, Y . Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt- 3?” arXiv preprint arXiv:2101.06804, 2021

    2021 arXiv

  5. [16]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,

    Y . Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” arXiv preprint arXiv:2104.08786 , 2021

    2021 arXiv

  6. [17]

    Multilingual llms are better cross-lingual in-context learners with alignment,

    E. Tanwar, S. Dutta, M. Borthakur, and T. Chakraborty, “Multilingual llms are better cross-lingual in-context learners with alignment,” arXiv preprint arXiv:2305.05940, 2023

    2023 arXiv

  7. [18]

    Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator,

    H. J. Kim, H. Cho, J. Kim, T. Kim, K. M. Yoo, and S.-g. Lee, “Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator,” arXiv preprint arXiv:2206.08082, 2022

    2022 arXiv

  8. [19]

    Rethinking the role of demonstrations: What makes in-context learning work?

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 11 048– 11 064

    2022

  9. [20]

    What in-context learning “learns

    J. Pan, “What in-context learning “learns” in-context: Disentangling task recognition and task learning,” Ph.D. dissertation, Princeton University, 2023

    2023

  10. [21]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: a visual language model for few-shot learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 23 716–23 736, 2022

    2022

  11. [22]

    Openflamingo: An open-source framework for training large autoregressive vision-language models,

    A. Awadalla, I. Gao, J. Gardner, J. Hessel, Y . Hanafy, W. Zhu, K. Marathe, Y . Bitton, S. Gadre, S. Sagawa et al., “Openflamingo: An open-source framework for training large autoregressive vision-language models,” arXiv preprint arXiv:2308.01390 , 2023

    2023 arXiv

  12. [24]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592 , 2023

    2023 arXiv

  13. [25]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  14. [26]

    Exploiting cross-modal prediction and relation consistency for semisupervised image captioning,

    Y . Yang, H. Wei, H. Zhu, D. Yu, H. Xiong, and J. Yang, “Exploiting cross-modal prediction and relation consistency for semisupervised image captioning,” IEEE Transactions on Cybernetics , vol. 54, no. 2, pp. 890–902, 2024

    2024

  15. [27]

    Semi- supervised multi-modal clustering and classification with incomplete modalities,

    Y . Yang, D. Zhan, Y . Wu, Z. Liu, H. Xiong, and Y . Jiang, “Semi- supervised multi-modal clustering and classification with incomplete modalities,” IEEE Trans. Knowl. Data Eng. , vol. 33, no. 2, pp. 682– 695, 2021

    2021

  16. [28]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents,

    H. Laurenc ¸on, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karamcheti, A. Rush, D. Kiela et al., “Obelics: An open web-scale filtered dataset of interleaved image-text documents,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  17. [29]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer,...

    2014

  18. [30]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y . Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Making the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 6904–6913

    2017

  19. [31]

    Ok-vqa: A visual question answering benchmark requiring external knowledge,

    K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi, “Ok-vqa: A visual question answering benchmark requiring external knowledge,” in Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, 2019, pp. 3195–3204

    2019

  20. [32]

    Obelics: An open web-scale filtered dataset of interleaved image- text documents,

    H. Laurenc ¸on, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karamcheti, A. Rush, D. Kiela et al. , “Obelics: An open web-scale filtered dataset of interleaved image- text documents,” Advances in Neural Information Processing Systems , vol. 36, pp. 71...

    2023

  21. [33]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020

    2001 arXiv

  22. [34]

    Deciphering the impact of pretraining data on large language models through machine unlearning,

    Y . Zhao, L. Du, X. Ding, K. Xiong, Z. Sun, J. Shi, T. Liu, and B. Qin, “Deciphering the impact of pretraining data on large language models through machine unlearning,” arXiv preprint arXiv:2402.11537 , 2024

    2024 arXiv

  23. [35]

    Em- powering llms with logical reasoning: A comprehensive survey,

    F. Cheng, H. Li, F. Liu, R. van Rooij, K. Zhang, and Z. Lin, “Em- powering llms with logical reasoning: A comprehensive survey,” arXiv preprint arXiv:2502.15652, 2025

    2025 arXiv

  24. [36]

    Au- tomatic text summarization: A comprehensive survey,

    W. S. El-Kassas, C. R. Salama, A. A. Rafea, and H. K. Mohamed, “Au- tomatic text summarization: A comprehensive survey,” Expert systems with applications, vol. 165, p. 113679, 2021

    2021

  25. [37]

    The question answering systems: A survey,

    A. M. N. Allam and M. H. Haggag, “The question answering systems: A survey,” International Journal of Research and Reviews in Information Sciences (IJRRIS), vol. 2, no. 3, 2012

    2012

  26. [38]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2015, pp. 4566–4575

    2015

  27. [39]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  28. [40]

    Object hallucination in image captioning,

    A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko, “Object hallucination in image captioning,” arXiv preprint arXiv:1809.02156, 2018

    2018 arXiv

  29. [41]

    Exploring diverse in-context configurations for image captioning,

    X. Yang, Y . Wu, M. Yang, H. Chen, and X. Geng, “Exploring diverse in-context configurations for image captioning,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  30. [42]

    Gpt-4 technical report. arxiv 2303.08774,

    R. OpenAI, “Gpt-4 technical report. arxiv 2303.08774,” View in Article, vol. 2, 2023

    2023 arXiv

  31. [43]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog , vol. 1, no. 8, p. 9, 2019

    2019

  32. [44]

    Scaling language- image pre-training via masking,

    Y . Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He, “Scaling language- image pre-training via masking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 23 390–23 400

    2023

  33. [45]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,

    J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural information processing systems , vol. 32, 2019

    2019

  34. [46]

    Uniter: Universal image-text representation learning,

    Y .-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y . Cheng, and J. Liu, “Uniter: Universal image-text representation learning,” in European conference on computer vision . Springer, 2020, pp. 104– 120

    2020

  35. [47]

    Oscar: Object-semantics aligned pre-training for vision-language tasks,

    X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei et al., “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX

    2020

  36. [48]

    Springer, 2020, pp. 121–137

    2020

  37. [49]

    mplug: Effective and efficient vision-language learning by cross-modal skip-connections,

    C. Li, H. Xu, J. Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. da Cao, J. Zhang, S. Huang, F. Huang, J. Zhou, and L. Si, “mplug: Effective and efficient vision-language learning by cross-modal skip-connections,” in Conference on Empirical Methods in Natural Language ...

    2022

  38. [50]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International Conference on Machine Learning . PMLR, 2022, pp. 12 888–12 900

    2022

  39. [51]

    Coca: Contrastive captioners are image-text foundation models,

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation models,” arXiv preprint arXiv:2205.01917, 2022

    2022 arXiv

  40. [52]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  41. [53]

    Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,” arXiv preprint arXiv:1804.07461 , 2018

    2018 arXiv

  42. [54]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  43. [55]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  44. [56]

    Diverse demonstrations improve in- context compositional generalization,

    I. Levy, B. Bogin, and J. Berant, “Diverse demonstrations improve in- context compositional generalization,” arXiv preprint arXiv:2212.06800, 2022

    2022 arXiv

  45. [57]

    In-context learning with iterative demonstration selection,

    C. Qin, A. Zhang, A. Dagar, and W. Ye, “In-context learning with iterative demonstration selection,” ArXiv, vol. abs/2310.09881,

    arXiv

  46. [58]

    How do in-context examples affect compositional generalization?

    S. An, Z. Lin, Q. Fu, B. Chen, N. Zheng, J.-G. Lou, and D. Zhang, “How do in-context examples affect compositional generalization?” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023, pp. 11 027– 11 052

    2023

  47. [59]

    What makes good in-context examples for gpt-3?

    J. Liu, D. Shen, Y . Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt-3?” DeeLIO 2022, p. 100, 2022. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15

    2022

  48. [60]

    Z- icl: Zero-shot in-context learning with pseudo-demonstrations,

    X. Lyu, S. Min, I. Beltagy, L. Zettlemoyer, and H. Hajishirzi, “Z- icl: Zero-shot in-context learning with pseudo-demonstrations,” arXiv preprint arXiv:2212.09865, 2022

    2022 arXiv

  49. [61]

    Active example selection for in-context learning,

    Y . Zhang, S. Feng, and C. Tan, “Active example selection for in-context learning,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , 2022, pp. 9134–9148

    2022

  50. [62]

    Self-adaptive in-context learning: An information compression perspective for in-context example selec- tion and ordering,

    Z. Wu, Y . Wang, J. Ye, and L. Kong, “Self-adaptive in-context learning: An information compression perspective for in-context example selec- tion and ordering,” arXiv preprint arXiv:2212.10375 , 2022

    2022 arXiv

  51. [63]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,

    Y . Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) ,...

    2022

  52. [64]

    How does the textual information affect the retrieval of multimodal in-context learning?

    Y . Luo, Z. Zheng, Z. Zhu, and Y . You, “How does the textual information affect the retrieval of multimodal in-context learning?” arXiv preprint arXiv:2404.12866, 2024

    2024 arXiv

  53. [65]

    From introspection to best practices: Principled analysis of demonstrations in multimodal in-context learning,

    N. Xu, F. Wang, S. Zhang, H. Poon, and M. Chen, “From introspection to best practices: Principled analysis of demonstrations in multimodal in-context learning,” arXiv preprint arXiv:2407.00902 , 2024

    2024 arXiv

  54. [66]

    Icd- lm: Configuring vision-language in-context demonstrations by language modeling,

    Y . Peng, X. Yang, H. Ma, S. Xu, C. Zhang, Y . Han, and H. Zhang, “Icd- lm: Configuring vision-language in-context demonstrations by language modeling,” arXiv preprint arXiv:2312.10104 , 2023

    2023 arXiv

  55. [67]

    How to configure good in-context sequence for visual question answering,

    L. Li, J. Peng, H. Chen, C. Gao, and X. Yang, “How to configure good in-context sequence for visual question answering,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 710–26 720

    2024

  56. [68]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763

    2021

  57. [69]

    What makes multimodal in-context learning work?

    F. B. Baldassini, M. Shukor, M. Cord, L. Soulier, and B. Piwowarski, “What makes multimodal in-context learning work?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 1539–1550

    2024

  58. [70]

    Why larger language models do in-context learning differently?

    Z. Shi, J. Wei, Z. Xu, and Y . Liang, “Why larger language models do in-context learning differently?” in R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models , 2023

    2023

  59. [71]

    An explanation of in-context learning as implicit bayesian inference,

    S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, “An explanation of in-context learning as implicit bayesian inference,” in International Conference on Learning Representations , 2021

    2021

  60. [72]

    Revealing the dark secrets of bert,

    O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky, “Revealing the dark secrets of bert,” arXiv preprint arXiv:1908.08593 , 2019

    1908 arXiv

  61. [73]

    Label words are anchors: An information flow perspective for understanding in-context learning,

    L. Wang, L. Li, D. Dai, D. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun, “Label words are anchors: An information flow perspective for understanding in-context learning,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , 2023, pp. 9840–9855

    2023

  62. [74]

    Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection- allocation,

    Q. Huang, X. Dong, P. Zhang, B. Wang, C. He, J. Wang, D. Lin, W. Zhang, and N. Yu, “Opera: Alleviating hallucination in multi- modal large language models via over-trust penalty and retrospection- allocation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pa...

    2024

  63. [75]

    Label words are anchors: An information flow perspective for understanding in-context learning,

    L. Wang, L. Li, D. Dai, D. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun, “Label words are anchors: An information flow perspective for understanding in-context learning,”arXiv preprint arXiv:2305.14160, 2023

    2023 arXiv

  64. [76]

    Sampleattention: Near-lossless accel- eration of long context llm inference with adaptive structured sparse attention,

    Q. Zhu, J. Duan, C. Chen, S. Liu, X. Li, G. Feng, X. Lv, H. Cao, X. Chuanfu, X. Zhang et al. , “Sampleattention: Near-lossless accel- eration of long context llm inference with adaptive structured sparse attention,” arXiv preprint arXiv:2406.15486 , 2024

    2024 arXiv

  65. [77]

    Attention-driven reasoning: Un- locking the potential of large language models,

    B. Liao and D. Vasconcellos Vargas, “Attention-driven reasoning: Un- locking the potential of large language models,” arXiv e-prints , pp. arXiv–2403, 2024

    2024

  66. [78]

    Deep visual-semantic alignments for generating image descriptions,

    A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3128–3137

    2015

  67. [79]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,

    L. Chen, H. Zhao, T. Liu, S. Bai, J. Lin, C. Zhou, and B. Chang, “An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models,” in European Conference on Computer Vision . Springer, 2024, pp. 19–35

    2024

  68. [80]

    Multimodal c4: An open, billion-scale corpus of images interleaved with text,

    W. Zhu, J. Hessel, A. Awadalla, S. Y . Gadre, J. Dodge, A. Fang, Y . Yu, L. Schmidt, W. Y . Wang, and Y . Choi, “Multimodal c4: An open, billion-scale corpus of images interleaved with text,” Advances in Neural Information Processing Systems , vol. 36, pp. 8958–8974, 2023

    2023

  69. [81]

    Laion- 5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsmanet al., “Laion- 5b: An open large-scale dataset for training next generation image-text models,” Advances in neural information processing systems , vol. 35, pp....

    2022

  70. [82]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” Advances in neural information processing systems, vol. 36, pp. 21 702–21 720, 2023

    2023

  71. [83]

    Model tells you what to discard: Adaptive kv cache compression for llms,

    S. Ge, Y . Zhang, L. Liu, M. Zhang, J. Han, and J. Gao, “Model tells you what to discard: Adaptive kv cache compression for llms,” arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  72. [84]

    Efficient stream- ing language models with attention sinks, 2024,

    G. Xiao, Y . Tian, B. Chen, S. Han, and M. Lewis, “Efficient stream- ing language models with attention sinks, 2024,” URL https://arxiv. org/abs/2309.17453, p. 1, 2024

    2024 arXiv

  73. [85]

    Prune and tune: Improv- ing efficient pruning techniques for massive language models,

    A. Syed, P. H. Guo, and V . Sundarapandiyan, “Prune and tune: Improv- ing efficient pruning techniques for massive language models,” 2023

    2023

  74. [86]

    Structured prompt- ing: Scaling in-context learning to 1,000 examples,

    Y . Hao, Y . Sun, L. Dong, Z. Han, Y . Gu, and F. Wei, “Structured prompt- ing: Scaling in-context learning to 1,000 examples,” arXiv preprint arXiv:2212.06713, 2022

    2022 arXiv

  75. [87]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” in International Conference on Machine Learning. PMLR, 2023, pp. 10 323–10 337

    2023

  76. [88]

    In-context learning with long-context models: An in-depth exploration,

    A. Bertsch, M. Ivgi, E. Xiao, U. Alon, J. Berant, M. R. Gormley, and G. Neubig, “In-context learning with long-context models: An in-depth exploration,” arXiv preprint arXiv:2405.00200 , 2024. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 16 VI. B IOGRAPHY SECTION ...

    2024 arXiv

  77. [89]

    Par- allel context windows for large language models,

    N. Ratner, Y . Levine, Y . Belinkov, O. Ram, I. Magar, O. Abend, E. Karpas, A. Shashua, K. Leyton-Brown, and Y . Shoham, “Par- allel context windows for large language models,” arXiv preprint arXiv:2212.10947, 2022

    2022 arXiv

  78. [91]

    Jingze Zhu received the B.Eng

    His research interests mainly include video understanding, in-context learning, machine unlearn- ing, and image editing. Jingze Zhu received the B.Eng. degree from the School of Computer Science and Engineering,, Southeast University in 2024. He is currently pur- suing the M.E...

    2024

  79. [2023]

    Available: https://api.semanticscholar.org/CorpusID: 264146526

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 264146526

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.