Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Efficient Multi-modal Long Context Learning for Training-free Adaptation

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EMLoC, a training-free compression method, cuts 20 multimodal demonstrations from 11,338 to 2,600 average tokens—a 77% reduction—while matching or beating full-context accuracy on most benchmarks.

desk verdict A promising adaptive KV-pruning method for multimodal in-context learning, with a misleading headline metric and a few reporting gaps; the core idea deserves a serious referee. read the letter →

arxiv 2505.19812 v1 pith:T2CN2VPQ submitted 2025-05-26 cs.CV

Zehong Ma , Shiliang Zhang , Longhui Wei , Qi Tian This is my paper
classification cs.CV
keywords training-freeadaptationmulti-modallongcontextlearningKVcachecompressionlayer-wiseadaptivepruningJensen-Shannondivergencein-contextvision-languagemodelschunk-wise
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a multimodal large language model can be adapted to a new task without any training by feeding it demonstration examples and then compressing those examples into a compact task-specific memory. The proposed method, EMLoC, splits the long demonstration context into chunks and prunes each chunk's key-value cache layer by layer, guided by answer-token attention and a Jensen-Shannon divergence check. On six vision-language benchmarks, the compression reduces the average context length from 11,338 to 2,600 tokens with 20 examples—a 77% reduction—while matching or beating the full-context baseline on most benchmarks. A reader should care because it offers a training-free, plug-and-play route to few-shot adaptation at a fraction of the inference cost.

What carries the argument

The central object is the compressed task-specific memory $\mathcal{M}$, the surviving KV cache produced from the demonstration chunks. Chunk-wise compression splits the long context into $K$ chunks (default 1.6k tokens) so each fits on one GPU; within each chunk, layer-wise adaptive pruning retains, per layer $l$, the top $r \times S$ tokens by accumulated answer-token attention $\beta^l_j$, trying retention ratios from an ascending set until the Jensen-Shannon divergence—a measure of how much two probability distributions differ—between original probabilities $p^{\mathrm{ori}}$ and iteratively pruned probabilities $p^{\mathrm{iter}}$ stays below $\delta$. The supporting theoretical result is a JS-distance triangle inequality giving a global upper bound that scales linearly with chunk count and with $\sqrt{\delta}$, which is what lets the paper treat the local pruning threshold as the single hyperparameter controlling the accuracy/compression trade-off.

What would settle it

Take ImageNet100 with 200 demonstrations at the default threshold $\delta = 0.005$; the claim predicts that the compressed memory keeps accuracy at least as high as the full-context baseline score of 62.6. If accuracy on the full 5,000-image validation set falls below 60.0 while the JS-divergence check still passes, the pruning proxy is not faithful. The same test can be run on any benchmark where EMLoC's compressed memory is compared against full-context MLoC.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that long multimodal demonstration contexts are highly redundant and that the redundancy can be removed greedily without retraining. EMLoC scores every token in a chunk's KV cache by the attention it receives from answer tokens, then prunes the least-important tokens in each layer from top to bottom, trying successively larger retention ratios until the Jensen-Shannon divergence between the original and pruned answer-token output distributions falls below a threshold $\delta$. The final compressed memory preserves task accuracy, and on most benchmarks it matches or slightly exceeds full-context performance because pruning removes irrelevant background noise. The paper derives a linear upper bound on the global information loss, $D_{\mathrm{JS}}(P^{\mathcal{D}}_{\mathcal{M}}, P^{\mathcal{D}}_{\mathcal{M}^K}) \le (K-1)\sqrt{\delta} + \epsilon$, to justify using the local per-layer constraint as a global control knob.

Load-bearing premise

EMLoC rests on the premise that answer tokens are a faithful observation window—tokens they attend to are the ones the task needs—and that keeping the Jensen-Shannon divergence of answer-token probabilities below $\delta$ always preserves task accuracy.

Editorial extensions

If this is right

  • With 20 demonstration examples, EMLoC uses 2,600 tokens on average instead of 11,338, a 77% reduction, and on most benchmarks it matches or exceeds full-context accuracy.
  • On ImageNet100 with 200 examples, EMLoC reaches 63.7% accuracy with 3,643 tokens, beating the full-memory MLoC score of 62.6% at 16,264 tokens.
  • Inference cost falls with the context: on ImageNet100, LLM FLOPs drop from 1.76T to 1.35T and total inference time from 1,866s to 1,107s.
  • Adaptation remains training-free and fast: 144s for EMLoC versus 234s for LoRA and 820s for full fine-tuning, with average accuracy 54.9 versus 54.7 for LoRA across ImageNet100, MME-RW, and OK-VQA.
  • The layer-wise pruning profiles show that layer importance is not a fixed pyramid; layers 4, 8, and 14 of Qwen2-VL retain markedly more tokens than their neighbors, so fixed-ratio pruning loses accuracy where adaptive pruning does not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the JS-divergence proxy generalizes beyond the six tested benchmarks, the same compression could be applied to the demonstrations retrieved by an online retrieval method, turning a five-hour, 43G-memory retrieval-augmented inference procedure into one that takes minutes and 18G.
  • The observed stability of the global JS divergence as the chunk count $K$ grows suggests inter-chunk dependencies are weak in practice; a testable extension is to push $K$ much higher and check whether the effective bound $\Delta \le \gamma\delta$ with $\gamma \le 2$ continues to hold, which would let memory usage scale down almost arbitrarily with chunk size.
  • Since image tokens make up the majority of pruned tokens, a natural follow-up is to assign modality-specific thresholds or retention ratios; one could test whether treating visual and text tokens separately improves the compression/accuracy trade-off beyond the single-threshold version.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EMLoC, a training-free method for adapting multimodal large language models to downstream tasks by providing in-context demonstration examples. To reduce the computational cost of long contexts, EMLoC partitions demonstrations into chunks, computes a KV cache for each chunk, and then prunes tokens layer-wise using attention-based importance scores under a Jensen-Shannon divergence constraint. The compact memory is then used for inference. The authors report experiments on six vision-language benchmarks with Qwen2-VL, claiming a 77% average context-length reduction at 20 examples without sacrificing accuracy, and they provide a theoretical bound on information loss in terms of the local JS threshold and the number of chunks.

Significance. If the efficiency claims hold, EMLoC is a practical contribution: it offers training-free adaptation with modest memory and latency overhead at adaptation time, and it outperforms static KV-cache pruning baselines such as SnapKV/H2O and PyramidKV on the tested benchmarks. The paper also includes useful ablations on the observation window, retention ratios, chunk sizes, and JS thresholds, and the code is publicly released. However, the central efficiency metric conflates raw prompt length with post-pruning memory size, the theoretical bound rests on unproven or explicitly empirical inequalities, and the claimed performance parity is contradicted by one benchmark and unsupported by error bars. These issues need to be addressed before the main claims can be accepted.

major comments (4)
  1. [Section 4.2 and Eq. (7)] The headline claim that EMLoC reduces the average context length from 11338 to 2600 (a 77% reduction) compares the raw prompt length of MLoC with the post-pruning KV-memory length of EMLoC. These are not the same quantity: during adaptation, EMLoC still performs a forward pass over the full chunked context in Eq. (7) to compute attention weights and hidden states, and only the cached KV pairs are pruned. The paper does not report per-benchmark inference FLOPs or time for the 20-example setting, so the 77% figure does not directly translate into a 77% reduction in end-to-end inference cost. The authors should either report the actual input tokens processed during both phases or clearly define the metric as `context length at generation time after compression`, and provide FLOPs/time for the headline setting.
  2. [Table 1, YouCook2 row] The claim that EMLoC achieves performance on par with or superior to MLoC with full memory is contradicted by the YouCook2 results: MLoC obtains 108.8 with 20 examples, while EMLoC obtains 102.0, a 6.8-point drop. This is not a small difference, and it is not discussed in the text. The statement in Section 4.2 that EMLoC 'surpasses MLoC with full memory in most benchmarks' is technically true only if 'most' is read as five out of six, but the unqualified phrase 'without sacrificing performance' is not supported by the table. The authors should add per-task discussion, explain why YouCook2 degrades, and qualify the performance claim accordingly.
  3. [Section 3.3, Eqs. (14)-(16)] The theoretical upper bound in Eq. (16) relies on two inequalities that are not established. Eq. (14) asserts that adding chunks from the full memory monotonically reduces JS divergence; this is a plausible heuristic but is not proved and is not generally true for arbitrary conditional distributions. Eq. (15) is explicitly labeled 'empirically assume', and its direction is not obvious: the left-hand side uses the full demonstration set D while the right-hand side uses only the chunk-specific demonstrations D_k, and there is no reason that conditioning on more data must reduce the JS distance. Since the claimed linear bound is a stated contribution, the authors should either prove these inequalities under explicit assumptions or reframe the analysis as a heuristic motivation and validate Eq. (15) empirically on the evaluated benchmarks.
  4. [All experimental tables (Tables 1-3, 5-8)] All reported numbers appear to come from a single run with no error bars or multiple seeds. Given that the 'without sacrificing performance' claim rests on differences of 0.1-1.1 points on several benchmarks, these differences are within the range of normal seed-to-seed variability for MLLM evaluation. The authors should provide standard deviations over at least three runs, or otherwise justify that the comparisons are statistically meaningful. This is particularly important for benchmarks like IllusionVQA where EMLoC and MLoC are exactly tied (40.9 vs. 40.9).
minor comments (5)
  1. [Section 4.2] The text references 'Table 4.2' when comparing with other multimodal ICL methods; this should be 'Table 3'.
  2. [Section 4.2, Table 2 discussion] The phrase 'compresses the context by nearly a quarter' should be 'compresses the context to nearly a quarter of its original length' (or 'by about 78%'), since 3643/16264 is 22.4%.
  3. [Figure 2 and Table 8] There are minor typos: 'framewrk' in the Figure 2 caption and 'Abalation' in the Table 8 caption should be corrected.
  4. [Appendix C.4] The MedXpertQA dataset is introduced in the task-disparity analysis without a reference or a description of its construction and evaluation protocol; please add details so the experiment is reproducible.
  5. [Figure 3] The axes of Figure 3 are not clearly labeled in the manuscript text; please add explicit axis labels (e.g., JS threshold delta on the x-axis, accuracy/context length on the y-axes) and a legend.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the compression method is judged by held-out benchmark accuracy, and the theoretical bound is an explicitly conditional derivation, not a renamed input.

full rationale

EMLoC's main claims—compression ratio, accuracy, and inference cost—are validated on external benchmarks (ImageNet100, ScreenSpot, MME-RW, IllusionVQA, OK-VQA, YouCook2, VideoMME) rather than by the method's own optimization objective. The pruning algorithm minimizes retained tokens subject to a local Jensen-Shannon constraint, but the paper never equates that constraint with task accuracy; accuracy is reported separately in Tables 1, 3, 4, and 5, and the compression-efficiency numbers are measurements of the produced memory, not fitted targets. The theoretical global bound in Eq. (16) is a conditional consequence of explicitly stated assumptions (Eqs. (14)-(15), the latter labeled 'empirically assume'), so it is not a hidden reuse of the conclusion; Appendix D candidly replaces the linear-in-K worst-case bound with an empirically fitted Δ≤γδ, and this admission prevents any claim that the empirical trend was derived from the input. Co-authored citations (InternVL, Pink, LocLLM, MME-RW) appear only as background/model/dataset references and are not load-bearing for EMLoC's mechanism. The only self-referential element is the Figure 4 JS-divergence diagnostic, which measures the same quantity the algorithm is constrained to minimize, but it is illustrative and corroborated by independent accuracy comparisons, so it does not make the central derivation circular. The paper's headline '77% reduction' compares a full-prompt token count with a post-pruning memory size, a metric-definition concern rather than a circularity, and it does not affect this verdict.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method depends on a small set of hyperparameters (delta, retention ratios, chunk size) that are selected by hand or by empirical observation. The theoretical analysis adds two unproven assumptions (Eq. 14 and 15) and ultimately a fitted constant in the appendix. No new ontological entities are introduced.

free parameters (5)
  • JS divergence threshold delta = 0.005
    Set as default to balance compression and accuracy; Figure 3 and Table 11 show sensitivity, and the choice is made by hand after observing benchmark performance.
  • Retention ratio set R = [0.1, 0.2, 0.5, 1.0]
    Greedy search over ratios; Table 7 and Appendix C.3 show the default selected after empirical comparison of accuracy and compression.
  • Chunk size = 1.6k tokens
    Default chunk length; Table 6 shows memory/accuracy trade-offs and 3.2k causes OOM, so this number is chosen based on hardware constraints and empirical stability.
  • Observation window = answer tokens only
    Chosen over question and image tokens based on the ablation in Table 8, where other windows reduce accuracy.
  • Empirical bound constant gamma = <= 2
    In Appendix D the authors replace the theoretical linear bound with Delta <= gamma * delta, fitting gamma to observed JS values across experiments.
assumptions (4)
  • domain assumption Demonstration examples and test data are from the same distribution (Eq. 11)
    This makes the probability distribution on test data equal to that on demonstrations, which is required for the theoretical analysis in Section 3.3.
  • ad hoc to paper Adding chunks from the full memory monotonically reduces JS divergence (Eq. 14)
    Stated as 'Consequently' without proof, it asserts that Mk-1 + Mk is closer to the full memory than Mk-1 is. This monotonicity may not hold in general and is not demonstrated.
  • ad hoc to paper JS distance on the full demonstration set is no larger than on chunk-specific demonstrations (Eq. 15)
    The paper explicitly labels this an 'empirically assume' step. It is used to close the global bound but can fail when chunks are not representative of the whole demonstration set.
  • domain assumption Attention of answer tokens is a valid proxy for token importance
    The pruning strategy in Eq. (8) relies on accumulated attention from answer tokens. It is validated only empirically in Table 8 and is not a proven property of the model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Multi-modal Long Context Learning for Training-free Adaptation." pith.science (2026). https://pith.science/paper/T2CN2VPQ

@misc{pith2026250519812,
  author       = {Pith},
  title        = {Pith review of: Efficient Multi-modal Long Context Learning for Training-free Adaptation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2CN2VPQ}},
  note         = {Machine review of arXiv:2505.19812}
}
read the original abstract

Traditional approaches to adapting multi-modal large language models (MLLMs) to new tasks have relied heavily on fine-tuning. This paper introduces Efficient Multi-Modal Long Context Learning (EMLoC), a novel training-free alternative that embeds demonstration examples directly into the model input. EMLoC offers a more efficient, flexible, and scalable solution for task adaptation. Because extremely lengthy inputs introduce prohibitive computational and memory overhead, EMLoC contributes a chunk-wise compression mechanism combined with layer-wise adaptive pruning. It condenses long-context multimodal inputs into compact, task-specific memory representations. By adaptively pruning tokens at each layer under a Jensen-Shannon divergence constraint, our method achieves a dramatic reduction in inference complexity without sacrificing performance. This approach is the first to seamlessly integrate compression and pruning techniques for multi-modal long-context learning, offering a scalable and efficient solution for real-world applications. Extensive experiments on diverse vision-language benchmarks demonstrate that EMLoC achieves performance on par with or superior to naive long-context approaches. Our results highlight the potential of EMLoC as a groundbreaking framework for efficient and flexible adaptation of multi-modal models in resource-constrained environments. Codes are publicly available at https://github.com/Zehong-Ma/EMLoC.

Figures

Figures reproduced from arXiv: 2505.19812 by the authors.

Figure 1
Figure 1. The comparison between EMLoC and MLoC on Ima￾geNet100 with varying numbers of demonstration examples. With 200 examples, EMLoC achieves 4.4× context compression over vanilla MLoC without performance loss. It significantly outper￾forms MLoC with 50 examples using a similar context length. 1. Introduction In recent years, multi-modal large models (MLLMs) (Liu et al., 2024; Zhu et al., 2024; Bai et al., 2023) have achi… view at source ↗
Figure 2
Figure 2. (a) The overall framewrk of efficient multi-modal long-context learning. (b) Chunk-wise compression with layer-adaptive pruning, where pruning steps iteratively update output probabilities and are validated using a JS divergence check. Gray squares indicate pruned tokens, with red and green arrows representing failed and successful pruning steps, respectively. constraint, retaining only the most relevant tokens for … view at source ↗
Figure 3
Figure 3. Performance and context length trends of EMLoC on ImageNet100 with 200 examples across different δ values Appendix C.1. Results of different δ [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Remaining token number of EMLoC and PyramidKV in ImageNet100 with 200 demonstrations and MME-RW with 20 demonstrations. The corresponding JS divergence after pruning is also illustrated to demonstrate the advantage of EMLoC. 4.4. Visualization of Compression Remaining …
Figure 5
Figure 5. Figure 5: Distribution of pruned and reserved tokens. Distribution of Pruned and Reserved Tokens. This study serves as an initial exploration of multi-modal context com￾pression. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Trend of ∆ on various δ and K. Notably, [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 16 canonical work pages

  1. [1]

    M., Bohnet, B., Chan, S., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J

    Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Chan, S., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., et al. Many-shot in-context learning. arXiv preprint arXiv:2404.11018, 2024

    arXiv 2024

  2. [2]

    Flamingo : A visual language model for few-shot learning

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo : A visual language model for few-shot learning. NeurIPS, 35: 0 23716--23736, 2022

    work page 2022

  3. [3]

    Qwen-VL : A frontier large vision-language model with versatile abilities

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-VL : A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  4. [4]

    R., and Neubig, G

    Bertsch, A., Ivgi, M., Alon, U., Berant, J., Gormley, M. R., and Neubig, G. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.00200, 2024

    arXiv 2024

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    work page 2020

  6. [6]

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling

    Cai, Z., Zhang, Y., Gao, B., Liu, Y., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.02069, 2024

    arXiv 2024

  7. [7]

    Llavolta: Efficient multi-modal models via stage-wise visual context compression

    Chen, J., Ye, L., He, J., Wang, Z.-Y., Khashabi, D., and Yuille, A. Llavolta: Efficient multi-modal models via stage-wise visual context compression. arXiv preprint arXiv:2406.20092, 2024 a

    arXiv 2024

  8. [8]

    Shikra : Unleashing multimodal LLM's referential dialogue magic

    Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., and Zhao, R. Shikra : Unleashing multimodal LLM's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

Show all 48 references
  1. [9]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, pp.\ 19--35. Springer, 2024 b

    2024

  2. [10]

    How far are we to GPT-4V ? closing the gap to commercial multimodal models with open-source suites

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to GPT-4V ? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024 c

    2024 arXiv

  3. [11]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  5. [13]

    Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd

    Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Zhang, S., Duan, H., Zhang, W., Li, Y., et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512, 2024

    2024 arXiv

  6. [14]

    Model tells you what to discard: Adaptive kv cache compression for llms

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., and Gao, J. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023

    2023 arXiv

  7. [15]

    E2vpt: An effective and efficient approach for visual prompt tuning

    Han, C., Wang, Q., Cui, Y., Cao, Z., Wang, W., Qi, S., and Liu, D. E2vpt: An effective and efficient approach for visual prompt tuning. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.\ 17445--17456. IEEE Computer Society, 2023 a

    2023

  8. [16]

    Facing the elephant in the room: Visual prompt tuning or full finetuning? arXiv preprint arXiv:2401.12902, 2024

    Han, C., Wang, Q., Cui, Y., Wang, W., Huang, L., Qi, S., and Liu, D. Facing the elephant in the room: Visual prompt tuning or full finetuning? arXiv preprint arXiv:2401.12902, 2024

    2024 arXiv

  9. [17]

    How well does gpt-4v (ision) adapt to distribution shifts? a preliminary investigation

    Han, Z., Zhou, G., He, R., Wang, J., Xie, X., Wu, T., Yin, Y., Khan, S., Yao, L., Liu, T., et al. How well does gpt-4v (ision) adapt to distribution shifts? a preliminary investigation. arXiv preprint arXiv:2312.07424, 2023 b

    2023 arXiv

  10. [18]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  11. [19]

    J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  12. [20]

    MiniCPM : Unveiling the potential of small language models with scalable training strategies

    Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. MiniCPM : Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  13. [21]

    Multimodal task vectors enable many-shot multimodal in-context learning

    Huang, B., Mitra, C., Karlinsky, L., Arbelle, A., Darrell, T., and Herzig, R. Multimodal task vectors enable many-shot multimodal in-context learning. Advances in Neural Information Processing Systems, 37: 0 22124--22153, 2024 a

    2024

  14. [22]

    K., Patra, B., et al

    Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. Language is not all you need: Aligning perception with language models. NeurIPS, 36, 2024 b

    2024

  15. [23]

    Visual prompt tuning

    Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European conference on computer vision, pp.\ 709--727. Springer, 2022

    2022

  16. [24]

    A., Wang, J

    Jiang, Y., Irvin, J. A., Wang, J. H., Chaudhry, M. A., Chen, J. H., and Ng, A. Y. Many-shot in-context learning in multimodal foundation models. In ICML 2024 Workshop on In-Context Learning, 2024. URL https://openreview.net/forum?id=j2rKwWXdcz

    2024

  17. [25]

    In-context learning with many demonstration examples

    Li, M., Gong, S., Feng, J., Xu, Y., Zhang, J., Wu, Z., and Kong, L. In-context learning with many demonstration examples. arXiv preprint arXiv:2302.04931, 2023

    2023 arXiv

  18. [26]

    Snapkv: Llm knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  19. [27]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. NeurIPS, 36, 2024

    2024

  20. [28]

    DeepSeek-VL : Towards real-world vision-language understanding

    Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Sun, Y., et al. DeepSeek-VL : Towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024

    2024 arXiv

  21. [29]

    K., and Buehler, M

    Lu, W., Luu, R. K., and Buehler, M. J. Fine-tuning large language models for domain adaptation: Exploration of training strategies, scaling, model merging and synergistic capabilities. npj Computational Materials, 11 0 (1): 0 84, 2025

    2025

  22. [30]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp.\ 3195--3204, 2019

    2019

  23. [31]

    Kosmos-2 : Grounding multimodal large language models to the world

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2 : Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  24. [32]

    S., Sayeed, K

    Shahgir, H. S., Sayeed, K. S., Bhattacharjee, A., Ahmad, W. U., Dong, Y., and Shahriyar, R. Illusion VQA : A challenging optical illusion dataset for vision language models. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=7ysaJGs7zY

    2024

  25. [33]

    Locllm: Exploiting generalizable human keypoint localization via large language model

    Wang, D., Xuan, S., and Zhang, S. Locllm: Exploiting generalizable human keypoint localization via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024 a

    2024

  26. [34]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024 b

    2024 arXiv

  27. [35]

    M2pt: Multimodal prompt tuning for zero-shot instruction learning

    Wang, T., Liu, Y., Liang, J., Zhao, J., Cui, Y., Mao, Y., Nie, S., Liu, J., Feng, F., Xu, Z., et al. M2pt: Multimodal prompt tuning for zero-shot instruction learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 3723--3740, 2024 c

    2024

  28. [36]

    CogVLM : Visual expert for pretrained language models

    Wang, W., Lv, Q., Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L., Song, X., et al. CogVLM : Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  29. [37]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=NG7sS51zVF

    2024

  30. [38]

    Pink: Unveiling the power of referential comprehension for multi-modal llms

    Xuan, S., Guo, Q., Yang, M., and Zhang, S. Pink: Unveiling the power of referential comprehension for multi-modal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 13838--13848, June 2024

    2024

  31. [39]

    Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference

    Yang, D., Han, X., Gao, Y., Hu, Y., Zhang, S., and Zhao, H. Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 3258--3270, 2024

    2024

  32. [40]

    Yi: Open Foundation Models by 01.AI

    Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi: Open Foundation Models by 01.AI . arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  33. [41]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output

    Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., Guo, Q., Duan, H., Wang, B., Ouyang, L., et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024 a

    2024 arXiv

  34. [42]

    On the out-of-distribution generalization of multimodal large language models

    Zhang, X., Li, J., Chu, W., Hai, J., Xu, R., Yang, Y., Guan, S., Xu, J., and Cui, P. On the out-of-distribution generalization of multimodal large language models. arXiv preprint arXiv:2402.06599, 2024 b

    2024 arXiv

  35. [43]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024 c

    Zhang, Y.-F., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024 c

    2024 arXiv

  36. [44]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36: 0 34661--34710, 2023

    2023

  37. [45]

    Towards automatic learning of procedures from web instructional videos

    Zhou, L., Xu, C., and Corso, J. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  38. [46]

    MiniGPT-4 : Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4 : Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  39. [47]

    Mini GPT -4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Mini GPT -4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1tZbq88f27

    2024

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.