Pith. sign in

REVIEW 3 major objections 6 minor 62 references

GIRAFFE: Design Choices for Extending the Context Length of Visual Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Giraffe extends Qwen2-VL to a 128K-token context with a curated data recipe, a dimension-aware position embedding, and hybrid-resolution inference, reaching open-source state of the art on long-video benchmarks and competitive parity with…

desk verdict A useful long-context VLM recipe with one unverifiable core formula: the M-RoPE++ schedule in Eq. 6 mixes a dimension index with wavelength ratios, so the method as printed is not well-defined. read the letter →

arxiv 2412.12735 v1 pith:JCOV3ZKO submitted 2024-12-17 cs.CV cs.AIcs.CL

Mukai Li , Lei Li , Shansan Gong , Qi Liu This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords contextlengthextensionvisuallanguagemodelsM-RoPE++rotarypositionembeddingvideounderstandinglong-contextbenchmarksinstructiontuninghybrid-resolutiontraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper works out a full pipeline for taking an existing visual language model and stretching its context window from 16K to 128K tokens. It claims that three design decisions together do this: a curated mix of short and long multimodal instruction data (ETVLM), a new rotary position embedding variant (M-RoPE++) that extrapolates the temporal axis while interpolating height and width, and a hybrid-resolution video encoding during inference. The resulting model, Giraffe, is reported to be state-of-the-art among similarly sized open-source long VLMs and competitive with GPT-4V on long-video benchmarks while keeping short-image performance. The paper also reports that the effective context length of VLMs, like that of LLMs, falls short of training length, and that M-RoPE++ extends that effective length further than position interpolation or NTK scaling.

What carries the argument

The central object is M-RoPE++, a piecewise re-scaling of the rotary frequencies in M-RoPE, the three-dimensional rotary position embedding used by Qwen2-VL. M-RoPE allocates 16x rotary blocks in a 2:3:3 ratio to temporal, height, and width indices; M-RoPE++ applies no scaling to the low-dimensional (temporal) blocks, a linear interpolation in the middle (height) block range, and full position interpolation to the high-dimensional (width) blocks. This asymmetry is what lets the model extrapolate to longer videos without blurring the order of nearby frames, and it is the mechanism that the paper credits for the effective-length gains.

What would settle it

Take the same Qwen2-VL extension recipe and swap M-RoPE++'s assignment, interpolating the temporal band while extrapolating height and width. If this swapped variant does not lose to M-RoPE++ on a long-video temporal-order benchmark, the paper's claim that temporal frequencies were already well covered is wrong.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the right way to extend the context of a VLM that uses M-RoPE is to treat the temporal, height, and width frequency bands differently: keep the temporal rotary base untouched (extrapolate), linearly interpolate the height band, and fully interpolate the width band. This piecewise scheme, M-RoPE++, gives longer effective length and better downstream video performance than direct extrapolation, PI, or NTK-aware scaling under the same training budget. Combined with a balanced data recipe and one-stage instruction tuning of the LLM backbone, it yields Giraffe, a 128K-context model that leads open-source long VLMs and approaches GPT-4V on VideoMME and LongVideoBench.

Load-bearing premise

The whole M-RoPE++ advantage rests on the assumption that Qwen2-VL's pre-training already covered the temporal frequency bands of the rotary embedding, so those bands can be extrapolated; if those frequencies were undertrained, the method's gains would come from something other than the coverage story.

Editorial extensions

If this is right

  • A VLM extension can be done in one stage: instruction-tune the LLM backbone on mixed data, with no separate text-context extension or visual alignment phase.
  • Data recipes for long-context VLMs should hold short instruction data to a substantial share; the paper's recommended 60% long-data ratio keeps long gains without sacrificing short benchmarks.
  • Hybrid-resolution inference (a high-resolution frame followed by compressed frames) delivers long-video gains at roughly half the visual tokens.
  • The effective-length fall-short phenomenon transfers from LLMs to VLMs: even after training to 128K, PI- and NTK-extended models stay effective to about 40K, while M-RoPE++ raises that ceiling.
  • With 1024 frames, Giraffe exceeds GPT-4V on some VideoMME and LongVideoBench segments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the temporal-coverage premise is transferable, M-RoPE++ should extend other rotary-split multimodal backbones, not just Qwen2-VL; a quick test would run the same piecewise scaling on any model with a t:h:w rotary split.
  • The paper's frame-resolution trade-off suggests a token-budget view of long-video understanding: at a fixed token count, high-resolution anchors plus low-resolution context beats uniform resolution, which implies adaptive frame sampling could push effective length further.
  • The ETVLM balance principle (short data as a floor, video data as the main lever) is plausible for other long-context multimodal tasks and can be tested by sweeping the ratio on a held-out task set.
  • The GPT-4V comparisons are on specific benchmark segments and frame counts; a head-to-head with controlled frame sampling and temporal-order probes would separate positioning gains from resolution gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a systematic study of how to extend the context length of visual language models to 128K tokens, using the Qwen-VL and Qwen2-VL families as backbones. It introduces three design components: the ETVLM data recipe (mixing long text, short visual instruction, interleaved image, and video instruction data), the M-RoPE++ position-embedding scheme (extrapolating the temporal rotary dimensions and interpolating the spatial ones), and hybrid-resolution video encoding (combining high-resolution key frames with compressed frames). The resulting GIRAFFE models are evaluated on single-image, multi-image, and long-video benchmarks, and the paper claims state-of-the-art performance among similarly sized open-source long VLMs and competitiveness with GPT-4V.

Significance. If the claims hold, the paper would provide a practical and unusually thorough recipe for converting an existing VLM into a long-context model without multi-stage alignment, and the public release of ETVLM would be a useful community asset. The experimental scope is a clear strength: data-ratio ablations, length ablations, position-embedding comparisons, training-strategy comparisons, and resolution/frame-count trade-offs are all reported on a common evaluation suite. The central claims are, however, tied to a position-embedding formula that is not correctly specified as printed, and the headline state-of-the-art comparison does not control for the number of input frames; both issues need to be resolved before the contribution is verifiable and fairly benchmarked.

major comments (3)
  1. [§3.2, Eq. (6)] Equation (6) is not a valid piecewise schedule as printed. On the middle branch, the factor (d − r_{5x})/(r_{2x} − r_{5x}) subtracts the wavelength-ratio value r_{5x} from the raw block index d. These are different quantities, so the ramp does not in general equal 1 at d = 2x and does not equal 1/s at d = 5x; the schedule is discontinuous at the boundaries of the interval 2x < d ≤ 5x. If the intended expression is (r_d − r_{5x})/(r_{2x} − r_{5x}), where r_d = L′/λ_d, the equation should be corrected. Because the claimed advantage of M-RoPE++ over PI, NTK, and direct extrapolation in Table 2 depends on this exact schedule, and because no code is available in v1, the current manuscript does not fully specify its main technical contribution and cannot be independently reproduced.
  2. [Table 6, §4.3] The headline comparison is confounded by frame count. In Table 6, GIRAFFE is evaluated with 768 or 1024 frames, while Qwen2-VL-7B is evaluated with 256 frames, LongVILA-8B with 256 frames, and LongVA-7B with 128 frames. Tables 4 and 5 show that increasing the frame count for GIRAFFE from 512 to 1024 improves the VideoMME-Long score by several points (for example, from 57.6 to 59.1 in Table 5 under the hybrid-resolution setting, and to 60.5 in the final model row). A matched-frame-count comparison at 256 frames is therefore needed to support the claim that GIRAFFE is state-of-the-art because of the extension method rather than because it is shown more frames. At minimum, the manuscript should report the same model at 256 frames and disclose the total visual token budget used per method.
  3. [§3.5, Table 5] The method section is ambiguous about what is actually trained. The heading and Finding 5 say "hybrid-resolution training," but the evidence in Table 5 consists of inference-time configurations (L,m,s), and the final row of Table 6 is labeled "w/ Hybrid-res train&inf" without describing the training-time counterpart. The manuscript does not state how high- and low-resolution frames are sampled during optimization, whether the full model or only the LLM backbone is updated under this schedule, or whether the same (L,m,s)=(4,240,3) setting is used for both training and inference. Without this information, hybrid-resolution training is not a reproducible contribution.
minor comments (6)
  1. [§3.2, Eq. (1) and Eq. (6)] The parameter x in the "16x-dimensional" M-RoPE matrix and in the segment boundaries 2x, 5x, and 8x is never explicitly defined in the main text; please state that x is the number of rotary blocks allocated per segment.
  2. [§3.1, Figure 4] The 60% accuracy threshold used to define effective length is introduced without justification; since claims such as "effective length is less than one-third of the training length" depend on this threshold, please report how the conclusions change when the threshold is varied.
  3. [Appendix B] The text says Su's blog recommends an optimal RoPE base of 4.9×10^6 and then describes this as closely approximating the selected base of 5×10^5, but these values differ by an order of magnitude; this appears to be a typo and should be corrected.
  4. [§3.2] The claim that temporal frequencies are already fully covered during pre-training, while spatial frequencies are not, is asserted without direct evidence; an analysis of frequency coverage or an ablation that interpolates the temporal band instead of extrapolating it would make the design rationale testable.
  5. [Table 2] The table header mixes frame-count columns for VideoMME and the image-count column for Visual Haystacks, so the entry "100" could be read as either; please relabel the columns to make clear that the last column is the number of Visual Haystack images.
  6. [Appendix C] The comparison between single-stage and progressive extending says both use the same number of training steps, but it does not state whether they also use the same total number of tokens or the same data mix; this should be clarified for the ablation to be interpretable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims rest on external benchmark comparisons; self-citations are data/reference only.

full rationale

The paper's central claims are benchmark-driven rather than definitional. M-RoPE++ is defined as a position-embedding schedule in Eq. 6 from the extension ratio s and rotary wavelengths, and its superiority over PI, NTK, and direct extrapolation is established empirically in Table 2 on VideoMME and Visual Haystack. Giraffe's state-of-the-art claim is supported by comparisons against external open-source and closed-source models in Tables 6-8. The ETVLM data recipe is built from external datasets, and the paper explicitly excludes MLVU from its test set to maintain evaluation fairness. There are self-citations, notably M3IT and the effective-length study, but these are used as data sources or as prior empirical observations, not as load-bearing uniqueness theorems or as substitutes for the paper's own experiments. The possible notational inconsistency in Eq. 6, where a dimension index d appears in the numerator of a ramp that otherwise uses wavelength-ratio values r, is a correctness and reproducibility concern rather than a circularity: the formula does not define the benchmark score in terms of itself, and the paper's conclusions are not forced by that formula alone. No fitted parameter is renamed as a prediction, and no central result reduces by construction to its own inputs. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on empirical assumptions about the pre-training coverage of M-RoPE frequency dimensions, the transferability of the LLM 'falls short' phenomenon to VLMs, the validity of the chosen benchmarks as measures of long-context ability, and the correctness of the ad hoc ramp in Eq. (6). None of these is proven in the paper; the last appears to contain a notation error as printed.

free parameters (5)
  • RoPE base for GIRAFFE-QwenVL = 500,000
    Selected via ablations in Appendix B, Table 9, following the Su (2023) recommendation for 128K context. This is a hand-chosen model hyperparameter that affects long-context scores.
  • ETVLM data composition ratios = 20% long text, 25% short visual instruction, 25% interleave, 30% video; 60% long-to-short ratio
    Chosen from ablations in Sec. 2.3, Figs. 2 and 3, to balance long and short task performance.
  • Hybrid-resolution group configuration (L,m,s) = (4, 240, 3)
    Default configuration selected from Table 5 ablation; controls frame grouping, high-res token count, and low-res compression.
  • Effective-length accuracy threshold = 60%
    Hand-chosen in Sec. 3.1 to define the effective context length from Visual Haystack retrieval accuracy; the reported effective length claims depend on this threshold.
  • Progressive extending context lengths = 8K, 32K, 64K, 128K for Qwen-VL; 64K then 128K for Qwen2-VL
    Training schedule chosen in Appendix C; progressive stages outperform single-stage in Table 10.
assumptions (4)
  • domain assumption M-RoPE allocates a 2:3:3 ratio to temporal, height, and width rotary dimensions, and the temporal dimension occupies high-frequency components (lower dimensions) that were sufficiently trained during pre-training.
    Used to justify extrapolation on temporal and interpolation on height/width in M-RoPE++ (Sec. 3.2, Eq. 6). No coverage evidence is provided.
  • domain assumption The effective context length of VLMs falls short of training length, analogous to LLMs.
    Motivated by citations (An et al., 2024b; Gao et al., 2024a) and observed in Fig. 4; foundational to the motivation for M-RoPE++.
  • domain assumption Benchmarks VideoMME, LongVideoBench, and Visual Haystacks, with the chosen frame counts and resolutions, are fair and representative measures of long-context VLM ability.
    The SOTA claim rests on these comparisons, but models are evaluated at different frame counts and resolutions, and no error bars are provided.
  • ad hoc to paper The ramp formula in Eq. (6) is a valid piecewise interpolation scheme.
    As printed, the formula mixes the dimension index d with the ratio r; its correctness is not derived and it appears dimensionally inconsistent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GIRAFFE: Design Choices for Extending the Context Length of Visual Language Models." pith.science (2026). https://pith.science/paper/JCOV3ZKO

@misc{pith2026241212735,
  author       = {Pith},
  title        = {Pith review of: GIRAFFE: Design Choices for Extending the Context Length of Visual Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCOV3ZKO}},
  note         = {Machine review of arXiv:2412.12735}
}
read the original abstract

Visual Language Models (VLMs) demonstrate impressive capabilities in processing multimodal inputs, yet applications such as visual agents, which require handling multiple images and high-resolution videos, demand enhanced long-range modeling. Moreover, existing open-source VLMs lack systematic exploration into extending their context length, and commercial models often provide limited details. To tackle this, we aim to establish an effective solution that enhances long context performance of VLMs while preserving their capacities in short context scenarios. Towards this goal, we make the best design choice through extensive experiment settings from data curation to context window extending and utilizing: (1) we analyze data sources and length distributions to construct ETVLM - a data recipe to balance the performance across scenarios; (2) we examine existing position extending methods, identify their limitations and propose M-RoPE++ as an enhanced approach; we also choose to solely instruction-tune the backbone with mixed-source data; (3) we discuss how to better utilize extended context windows and propose hybrid-resolution training. Built on the Qwen-VL series model, we propose Giraffe, which is effectively extended to 128K lengths. Evaluated on extensive long context VLM benchmarks such as VideoMME and Viusal Haystacks, our Giraffe achieves state-of-the-art performance among similarly sized open-source long VLMs and is competitive with commercial model GPT-4V. We will open-source the code, data, and models.

Figures

Figures reproduced from arXiv: 2412.12735 by the authors.

Figure 1
Figure 1. Pipeline of extending visual language models. We collect data from text, text-image pairs, and videos. We [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Performance of extending Qwen-VL with different data composition ratios. Data Length We categorize data into long data and short data based on whether their length ex￾ceeds 8K tokens. We investigate how different ratios of long and short data affect downstream per￾formance on both long-context and short-context tasks. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Results on visual haystack. The x-axis shows [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (1 more)
Figure 3
Figure 3. Figure 3: Performance on Qwen-VL trained with differ [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 11 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. 2024 a . http://arxiv.org/abs/2402.17463 Training-free long-context scaling of large language models

    arXiv 2024

  4. [4]

    Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. 2024 b . http://arxiv.org/abs/2410.18745 Why does the effective context length of llms fall short?

    arXiv 2024

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. http://arxiv.org/abs/2308.12966 Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond

    arXiv 2023

  6. [6]

    Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024. http://arxiv.org/abs/2401.18058 Longalign: A recipe for long context alignment of large language models

    arXiv 2024

  7. [7]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023 a . http://arxiv.org/abs/2311.12793 Sharegpt4v: Improving large multi-modal models with better captions

    arXiv 2023

  8. [8]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023 b . http://arxiv.org/abs/2306.15595 Extending context window of large language models via positional interpolation

    arXiv 2023

Show all 62 references
  1. [9]

    Yukang Chen, Shaozuo Yu, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023 c . Long alpaca: Long-context instruction-following models. https://github.com/dvlab-research/LongLoRA

    2023

  2. [10]

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821

    2024 arXiv

  3. [11]

    Tri Dao. 2024. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR)

    2024

  4. [12]

    Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. https://openreview.net/forum?id=H4DqfPSibmx Flashattention: Fast and memory-efficient exact attention with IO -awareness . In Advances in Neural Information Processing Systems

    2022

  5. [13]

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019. http://arxiv.org/abs/1812.03982 Slowfast networks for video recognition

    2019 arXiv

  6. [14]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394

    2023 arXiv

  7. [15]

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. 2024 a . Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075

    2024 arXiv

  8. [16]

    Smith, Wei-Chiu Ma, and Ranjay Krishna

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. 2024 b . http://arxiv.org/abs/2404.12390 Blink: Multimodal large language models can see but not perceive

    2024 arXiv

  9. [17]

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024 c . https://openreview.net/forum?id=TaAqeo7lUh Data engineering for scaling language models to 128k context . In Forty-first International Conference on Machine Learning

    2024

  10. [19]

    Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024 b . How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660

    2024

  11. [20]

    Gemini Team . 2024. http://arxiv.org/abs/2312.11805 Gemini: A family of highly capable multimodal models

    2024 arXiv

  12. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

    2022

  13. [22]

    Ku, Qian Liu, and Wenhu Chen

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen. 2024. Mantis: Interleaved multi-image instruction tuning. arXiv2405.01483

    2024 arXiv

  14. [23]

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024 a . http://arxiv.org/abs/2407.07895 Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    2024 arXiv

  15. [24]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 a . https://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . ArXiv preprint, abs/2301.12597

    2023 arXiv

  16. [25]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , vo...

    2022

  17. [26]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2024 b . http://arxiv.org/abs/2305.06355 Videochat: Chat-centric video understanding

    2024 arXiv

  18. [27]

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.775 Multimodal A r X iv: A dataset for improving scientific comprehension of large vision-language models . In Proceedings of the 62nd Annual ...

    2024 doi

  19. [28]

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. 2023 b . https://arxiv.org/abs/2306.04387 M ^3 IT : A large-scale dataset towards multi-modal multilingual instruction tuning . ArXiv pr...

    2023 arXiv

  20. [29]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2024. http://arxiv.org/abs/2311.10122 Video-llava: Learning united visual representation by alignment before projection

    2024 arXiv

  21. [30]

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. 2024 a . World model on million-length video and language with ringattention. arXiv preprint

    2024

  22. [31]

    Hao Liu, Matei Zaharia, and Pieter Abbeel. 2023 a . https://api.semanticscholar.org/CorpusID:263608461 Ring attention with blockwise transformers for near-infinite context . ArXiv, abs/2310.01889

    2023 arXiv

  23. [32]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023 b . https://arxiv.org/abs/2310.03744 Improved baselines with visual instruction tuning . arXiv preprint arXiv:2310.03744

    2023 arXiv

  24. [33]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 c . https://arxiv.org/abs/2304.08485 Visual instruction tuning . ArXiv preprint, abs/2304.08485

    2023 arXiv

  25. [34]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 b . http://arxiv.org/abs/2307.06281 Mmbench: Is your multi-modal model an all-around player?

    2024 arXiv

  26. [35]

    Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al. 2024 c . Mmdu: A multi-turn multi-image dialog understanding benchmark and instruction-tuning dataset for lvlms. arXiv preprint arXiv:2406.11833

    2024 arXiv

  27. [36]

    LocalLLaMA. 2023. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/ Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

    2023

  28. [37]

    OpenAI. 2023. https://openai.com/research/gpt-4v-system-card Gpt-4v(ision) system card . OpenAI Research

    2023

  29. [38]

    OpenAI. 2024. https://github.com/openai/openai-python/blob/main/chatml.md Chatml documents

    2024

  30. [39]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. http://arxiv.org/abs/2309.00071 Yarn: Efficient context window extension of large language models

    2023 arXiv

  31. [40]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

    2020

  32. [41]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  33. [42]

    Jianlin Su. 2023. https://kexue.fm/archives/10122 Extending llm context window beyond 2048 tokens

    2023

  34. [43]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. https://doi.org/10.1016/j.neucom.2023.127063 Roformer: Enhanced transformer with rotary position embedding . Neurocomput., 568(C)

    2024

  35. [44]

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. 2022. http://arxiv.org/abs/2212.10554 A length-extrapolatable transformer

    2022 arXiv

  36. [45]

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2023. https://arxiv.org/abs/2309.14525 Aligning large multimodal models with factually augmented rlhf . arXiv ...

    2023 arXiv

  37. [46]

    Llama Team. 2024. http://arxiv.org/abs/2407.21783 The llama 3 herd of models

    2024 arXiv

  38. [47]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . http://arxiv.org/abs/2409.12191 Qwen2...

    2024 arXiv

  39. [48]

    Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. 2024 b . http://arxiv.org/abs/2409.02889 Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture

    2024

  40. [49]

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. 2024 a . http://arxiv.org/abs/2407.15754 Longvideobench: A benchmark for long-context interleaved video-language understanding

    2024 arXiv

  41. [50]

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. 2024 b . Q-bench: A benchmark for general-purpose foundation models on low-level vision. In ICLR

    2024

  42. [51]

    Gonzalez, Trevor Darrell, and David M

    Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E. Gonzalez, Trevor Darrell, and David M. Chan. 2024 c . http://arxiv.org/abs/2407.13766 Visual haystacks: A vision-centric needle-in-a-haystack benchmark

    2024 arXiv

  43. [52]

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, and Maosong Sun. 2024 a . Infllm: Unveiling the intrinsic capacity of llms for understanding extremely long sequences with training-free memory. arXiv

    2024

  44. [53]

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024 b . https://openreview.net/forum?id=NG7sS51zVF Efficient streaming language models with attention sinks . In The Twelfth International Conference on Learning Representations

    2024

  45. [54]

    Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. 2024. http://arxiv.org/abs/2408.10188 Longvila: Scaling long-...

    2024 arXiv

  46. [55]

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. 2024 a . https://arxiv.org/abs/2406.16852 Long context transfer from language to vision . arXiv preprint arXiv:2406.16852

    2024 arXiv

  47. [56]

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024 b . http://arxiv.org/abs/2410.02713 Video instruction tuning with synthetic data

    2024 arXiv

  48. [57]

    Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. 2024. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264

    2024 arXiv

  49. [58]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. https://arxiv.org/abs/2304.10592 Minigpt-4: Enhancing vision-language understanding with advanced large language models . ArXiv preprint, abs/2304.10592

    2023 arXiv

  50. [59]

    Zhilin Zhu. 2023. Ring flash attention. https://github.com/zhuzilin/ring-flash-attention

    2023

  51. [60]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  52. [61]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  53. [62]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  54. [63]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.