Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Low-rank adaptation of the LLM during vision-language alignment outperforms full fine-tuning by preserving modality-specific visual structure.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 11:33 UTC pith:VKL566SM

load-bearing objection Useful PEFT-for-alignment result with a clean feature story (LS-curse, homogeneous tokens); the blanket “beats full FT” claim is softer than the structure-preserving evidence. the 3 major comments →

arxiv 2607.08194 v1 pith:VKL566SM submitted 2026-07-09 cs.CV

Dive Into the Implicit Biases of Low-rank Vision-language Alignment

Mingjia Shi , Shuo Wang , Xiaobo Wang , Sifan Zhou , Kai Wang , Tianyu Fu , Chenxu Zhao , Anyang Su
show 2 more authors
Ping Jiang Minghui Wu
This is my paper
classification cs.CV
keywords vision-language alignmentlow-rank adaptationLoRAimplicit biasLS-cursemultimodal modelsparameter-efficient fine-tuningvisual tokens
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models are usually aligned by fully updating the language model when a pretrained vision encoder is attached. This paper challenges that default. It shows that low-rank adapters on the LLM during alignment are cheaper and, from 1.4B to 14B parameters, usually better on perception, knowledge, reasoning, and hallucination benchmarks. The advantage is not only efficiency: low-rank updates keep visual tokens linearly separable and geometrically homogeneous, while full updates scramble some tokens—an effect the authors call the LS-curse—and fuse entity-level semantics too early. Two theorems cast this as a preference for flat, noise-robust subspaces. The result supports a progressive-fusion view: keep general visual structure during alignment and leave entity-specific mixing for instruction tuning.

Core claim

Applying low-rank adaptation to the LLM during vision-language alignment not only reduces cost but consistently outperforms full-parameter alignment on most benchmarks across 1.4B–14B scales, because it induces structure-preserving implicit biases—more conservative, less hallucinatory behavior; preservation of per-token linear separability of visual features; and homogeneous, stable manifolds—rather than the premature entity-level fusion that full updates produce.

What carries the argument

The LS-curse (full alignment’s disruption of per-token linear separability of visual features) plus Theorems 1 and 2: low-rank gradient flow preferentially reinforces noise-robust feature directions, and the steady-state low-rank parameters concentrate on flat-gradient, perturbation-robust subspaces.

Load-bearing premise

The headline gains partly rest on choosing the best low-rank operator and rank after a grid search, while full-parameter alignment is one method under the same trial budget, so the comparison can favor the more flexible low-rank family.

What would settle it

Run matched full-parameter versus single-operator low-rank (fixed rank, no operator selection) alignment with identical hyperparameter search budgets on the same data; if full-parameter then matches or beats low-rank on the same perception and POPE suite across scales, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Vision-language alignment can be treated as supervised fine-tuning rather than pretraining, making low-rank LLM updates the natural default.
  • Freezing the vision encoder and adapting only the LLM preserves more usable visual structure for later stages.
  • Progressive fusion becomes a design rule: retain modality-common structure in alignment; defer entity-specific fusion to instruction tuning.
  • Operator family (LoRA, LoHa, LoKr) and rank are first-class knobs that trade peak fine-grained separability for homogeneous preservation.
  • The same pattern holds across CLIP, DINOv2, and SigLIPv2 under frozen-encoder defaults, so the bias is not encoder-specific.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same structure-preserving bias may help explain why low-rank fine-tuning often reduces catastrophic forgetting in unimodal LLM adaptation.
  • If the LS-curse is the main failure mode of full alignment, cheap per-token linear-probe diagnostics could flag bad alignments before expensive instruction tuning.
  • A rank schedule that starts low and rises could implement progressive fusion inside one stage rather than across stages.
  • Once progressive fusion is the goal, connector design may matter less than the rank and placement of LLM-side updates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that vision–language alignment should be treated as supervised fine-tuning rather than full pretraining, and that applying low-rank adaptation (LoRA/LoHa/LoKr) to the LLM during this stage both cuts cost and improves downstream performance relative to full-parameter alignment across 1.4B–14B models. Empirically, low-rank alignment is linked to more conservative (lower Type-I) perception, preservation of per-token linear separability of visual features (avoiding an “LS-curse” under full updates), and more homogeneous token manifolds. The authors interpret this as progressive fusion: modality-common structure is retained at alignment and entity-level mixing is deferred. Two theorems under an unconstrained-feature model and noisy gradient flow attribute the bias to noise-robust feature directions and concentration on flat-gradient low-rank subspaces. Experiments span many ranks, three operator families, encoders (CLIP/DINOv2/SigLIP), freeze settings, and related ablations.

Significance. If the comparison is made fair and the structure-preserving account holds, the work is significant for multimodal training practice: it challenges the default of full LLM updates at alignment and supplies a concrete mechanistic story (LS preservation, manifold homogeneity, Type-I/II shifts) rather than only PEFT efficiency. Strengths include breadth (scales, operators, encoders, freeze/unfreeze, depth grids), explicit reporting of counter-cases (e.g., POPE/MMMU-Pro at some ranks), and an attempt to connect behavior, geometry, and optimization theory. The theorems are explanatory under standard idealized assumptions rather than quantitative certificates; their value is directional prediction of the observed biases. The progressive-fusion framing and the LS-curse observation are useful contributions even if headline accuracy gaps shrink under stricter protocols.

major comments (3)
  1. [Table 1; §2 training protocol; Tables 3–5] Table 1 (and §2 training protocol; Tables 3–5): the headline claim that low-rank “outperforms full-parameter alignment on most benchmarks” is load-bearing on an asymmetric protocol. Low-rank is reported as the best of LoRA/LoHa/LoKr after rank tuning (and “best out of 9” hyperparameter trials), while full alignment is a single method under the same trial budget. Operator choice is strongly rank- and scale-dependent (Table 3: LoHa r=128 MME-P 1079.7 vs full 236.6 on 13B, while other ranks/operators collapse). Please report fixed-operator, fixed-rank comparisons (and mean±std over seeds) as primary results, with best-of as secondary, so gains are not confounded with multi-operator selection.
  2. [Table 1; §4; Tables 5–7] Several full-parameter baselines look under-tuned relative to standard VLM practice (e.g., Table 1: 1.4B POPE 2.5, 2.7B MME-P 15.7; large absolute gaps that reverse or shrink under other settings in Tables 5–7). If full FT is poorly optimized, the performance narrative overstates an intrinsic low-rank advantage. Strengthen full baselines (LR schedules, weight decay, longer alignment, matched wall-clock or matched trainable-parameter budgets where appropriate) and state clearly when low-rank wins under a single fixed operator at matched compute.
  3. [§2.2–2.4; §3 Theorems 1–2; Appendix 2.1] §2.2–2.4 and §3: the structure-preserving story (Type-I reduction, per-token LS, angular coverage, manifolds) is cleaner than the multi-operator accuracy claim and does not require best-of selection, but the link from Theorems 1–2 (noise-weighted third-order smoothing; F_eff with log-det flatness and feature-noise penalties) to these measurements remains associational. The appendix already frames this carefully; the main text should (i) lead with fixed-operator geometric/behavioral results as the primary evidence for implicit bias, and (ii) state what would falsify the theory (e.g., low-rank runs that destroy per-token LS while still improving POPE/MME). Without that, the theorems risk reading as post-hoc rationalization of the performance table.
minor comments (5)
  1. [§2.3; Figure 4] Figure 4 and the ImageNet per-token LS analysis use a 1.4B model with a 2-layer connector; state more prominently that fine-grained probing is compute-limited to smaller models and that large-model claims rest mainly on end-task metrics.
  2. [§3.2 Theorem 2] Notation for F_eff(M) and eD(M) in Theorem 2 / Appendix 1.3 is dense; a short informal paragraph defining “flatness” and “feature-noise” penalties before the formal statement would help non-theory readers.
  3. [Table 1] Table 1 formatting (nested brackets for full vs low-rank) is hard to parse; separate columns for full, best low-rank, and Δ would reduce ambiguity.
  4. [Appendix Table 12; training receipt] Clarify default LoRA α=256 and why it is fixed while r is swept; α/r ratio can dominate effective scale and should be documented as a free parameter.
  5. [§3; §5] Related work on classical matrix-factorization implicit bias is well placed in §5 / Appendix 1.4; a one-sentence pointer in the main theory section would help readers who skip the appendix.

Circularity Check

0 steps flagged

No significant circularity: empirical biases measured on external probes/benchmarks; theorems are explanatory derivations under stated assumptions, not self-defining the observations.

full rationale

The paper's load-bearing claims separate cleanly into (i) empirical measurements of behavior (Type-I/II on POPE), per-token linear separability (ImageNet/STF-dogs/CUB probes), angular coverage, and manifolds, all on held-out external datasets independent of any fitted theoretical quantity, and (ii) Theorems 1–2, which start from the Unconstrained Feature Model, additive feature noise, LoRA/LoHa/LoKr gradient flows, and a Fokker–Planck/SDE approximation of noisy SGD, then derive an effective loss containing flatness (log-det gradient-noise) and noise-robustness (third-order feature-noise) penalties. These effective losses are not fitted to the probe accuracies or benchmark scores; they are presented only as a mechanistic explanation for the already-observed structure-preserving patterns. No parameter is fit to a subset of the target quantities and then re-labeled a prediction; no uniqueness theorem is imported from overlapping-author prior work to force the result; classical matrix-factorization implicit-bias citations are explicitly distinguished rather than smuggled as the present bias. The progressive-fusion narrative is interpretive glue between independent observations and the theorems, not a definitional loop. Consequently the derivation chain does not reduce by construction to its inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 3 invented entities

The empirical claim rests mainly on experimental protocol choices (which operator/rank wins, frozen encoder, minimal connector pipeline). The theoretical claim rests on standard PEFT parameterizations plus idealized analysis assumptions (UFM, additive feature noise, small-step noisy GD / Fokker–Planck steady state) that are not independently validated for large VLMs. Free parameters are the usual training knobs that select the reported “best” low-rank configuration.

free parameters (4)
  • low-rank rank r (and operator choice among LoRA/LoHa/LoKr) = typically 32/64/128; default LoRA r=128
    Headline tables often take the best operator and rank after search; performance is highly rank- and operator-dependent (Tables 3–5).
  • LoRA alpha = 256
    Fixed empirically in the training recipe rather than derived.
  • alignment learning rate and schedule = e.g. 1e-3 cosine for MobileVLM alignment
    Chosen by trial budget (“best out of 9”); ablations show scheduler sensitivity is limited but LR remains a free training knob.
  • vision-tower feature layer / alignment depth = default deep CLIP layer (e.g. 22) with ablations over {2,6,...,22}
    Depth grid shows strong model-dependent optima; default second/selected layer is a design choice that affects measured gains.
axioms (6)
  • domain assumption Unconstrained Feature Model: overparameterized network can treat features H as free variables for analyzing preferred directions.
    Stated in §3 as the analysis model; standard in neural-collapse-style work but not proven for the LLM+adapter stack used empirically.
  • domain assumption Feature noise is zero-mean with covariance Σ; third-order Taylor remainder is negligible for the gradient expectation.
    Theorem 1 expansion in §3.1 / Appendix 1.2; drives the noise-smoothing preference.
  • domain assumption Small-step SGD is well approximated by noisy gradient flow whose steady state is Gibbs-like with det(D) flatness correction (Fokker–Planck).
    Theorem 2 and Appendix steady-state derivation; idealized relative to large-batch multimodal training.
  • domain assumption Vision-language alignment is supervised fine-tuning atop pretrained LLM/ViT priors, so PEFT is the appropriate regime.
    Core framing in §1; motivates applying LoRA at alignment rather than treating the stage as pretraining.
  • standard math Einstein summation / standard multivariable calculus for gradients of LoRA, LoHa, LoKr maps.
    Appendix preliminary gradient cases 0–3.
  • ad hoc to paper Minimal MLP-adapter VLM pipeline without proprietary multi-layer visual injection is representative enough to study alignment-stage biases.
    Defended in Related Works and Appendix limitations; scopes all main claims away from Qwen3-VL-style deepstack recipes.
invented entities (3)
  • LS-curse independent evidence
    purpose: Name the empirical failure mode where full-parameter alignment destroys per-token linear separability of some visual tokens.
    Operationalized via token-wise linear probes on ImageNet/fine-grained sets; useful label, not an independent physical object.
  • Progressive fusion view no independent evidence
    purpose: Interpretive framework: keep modality-general structure in alignment; defer entity-level fusion to instruction tuning.
    Summarizes Obs.1–2 and geometry; not separately falsified outside the same experiments.
  • Effective low-rank loss F_eff(M) with log-det flatness and feature-noise penalties no independent evidence
    purpose: Encode Theorem 2 steady-state bias toward flat, robust subspaces.
    Derived under Fokker–Planck assumptions; not measured as a standalone quantity in experiments.

pith-pipeline@v1.1.0-grok45 · 29423 in / 4125 out tokens · 48652 ms · 2026-07-10T11:33:35.941523+00:00 · methodology

0 comments
read the original abstract

Vision-language alignment, the stage that bridges pretrained vision encoders and large language models, is widely treated as a form of pretraining requiring full-parameter updates. We challenge this view and investigate what happens when low-rank adaptation is applied to the LLM during this stage instead. We find that low-rank alignment not only reduces computational costs but also outperforms full-parameter alignment on most benchmarks. To understand this phenomenon, we systematically characterize the implicit biases introduced by low-rank adaptation during alignment. Empirically, we find that low-rank alignment shifts model behavior from hallucinatory to conservative and preserves per-token linear separability of visual features that full-parameter alignment disrupts, a phenomenon we term LS-curse. Geometrically, low rank aligned models exhibit more homogeneous and structurally stable visual representations, maintaining modality-specific knowledge rather than prematurely fusing entity-level semantics. Theoretically, we establish two theorems showing that low-rank alignment induces preferences for parameter subspaces with flat gradients and feature subspaces robust to perturbations, providing a principled explanation for the observed structure-preserving behavior. Extensive experiments cover ablation over 100 alignment configurations, three families of low-rank operators, and various rank, encoder, and other settings.

Figures

Figures reproduced from arXiv: 2607.08194 by Anyang Su, Chenxu Zhao, Kai Wang, Minghui Wu, Mingjia Shi, Ping Jiang, Shuo Wang, Sifan Zhou, Tianyu Fu, Xiaobo Wang.

Figure 1
Figure 1. Figure 1: Vision-language alignment framework (default): a minimum achievable VLM with a cross-modal adapter (MLP). How does low-rank alignment shape visual tokens?. misconception: unlike LLM pretraining, which builds linguistic representations from scratch, vision-language alignment operates atop pretrained knowledge and reasoning priors: it is, in fact, supervised fine-tuning, precisely the regime for which low-ra… view at source ↗
Figure 2
Figure 2. Figure 2: Operators for low-rank alignment: LoRA/LoHa/LoKr (Matrix, Hadamard and Kronecker Products). Training protocol. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Type -I and -II error rate: models and low￾rank operators. The results are on POPE, a visual perception hallucination benchmark [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Broken per-token linear separability of visual tokens, marked in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Full aligned vision tokens have larger cover￾age of each given input images in ImageNet. Each image’s feature angle coverage (i.e., minimal cosine ↓ :=mini,j cos(i, j), i, j ∈ {visualtokens|images}). This asymmetry directly sup￾ports the progressive fusion hypothesis: low-rank align￾ment retains the general knowl￾edge backbone while deferring entity-specific fusion to subse￾quent tuning stages. Representat… view at source ↗
Figure 6
Figure 6. Figure 6: The feature manifolds of each visual token, as depicted in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: CLIP, the vision tower of related VLMs, originally has great general and fine￾grained knowledge structures in each token. Notation CLIP means features in original CLIP of 336×336 resolutions and 14 patch. Notations full/LoRA mean features in each vision token from full-parameter/LoRA pretrained VLMs. Fine-grained classification datasets: STF-dogs and CUB [PITH_FULL_IMAGE:figures/full_fig_p035_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Depth preferences: the optimal alignment depth depends on the model scale, while patterns in a given model across similar benchmarks remain consistent. Horizontal axis: the vision-tower layer used during the feature alignment stage, sampled from the original 23-layer CLIP encoder at layers {2, 6, 10, 14, 18, 22}. Vertical axis: the vision￾tower layer used during the instruction fine-tuning stage, selected … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 57 canonical work pages · 18 internal anchors

  1. [1]

    NeurIPS35, 23716–23736 (2022)

    Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: A visual language model for few-shot learning. NeurIPS35, 23716–23736 (2022)

    work page 2022

  2. [2]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Arora, S., Cohen, N., Hu, W., Luo, Y.: Implicit regularization in deep matrix factorization. In: Advances in Neural Information Processing Systems (NeurIPS) (2019)

    work page 2019

  3. [3]

    Qwen3-VL Technical Report

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Transactions on Machine Learning Research (TMLR) (2024)

    Biderman, D., Portes, J., Gonzalez Ortiz, J.J., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., Cunningham, J.P.: LoRA learns less and forgets less. Transactions on Machine Learning Research (TMLR) (2024)

    work page 2024

  5. [5]

    In: Eu- ropean Conference on Computer Vision

    Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: Sharegpt4v: Improving large multi-modal models with better captions. In: Eu- ropean Conference on Computer Vision. pp. 370–387. Springer (2024)

    work page 2024

  6. [6]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24185–24198 (2024)

    work page 2024

  7. [7]

    See https://vicuna

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023)2(3), 6 (2023)

    work page 2023

  8. [8]

    MobileVLM V2: Faster and Stronger Baseline for Vision Language Model

    Chu, X., Qiao, L., Zhang, X., Xu, S., Wei, F., Yang, Y., Sun, X., Hu, Y., Lin, X., Zhang, B., et al.: Mobilevlm v2: Faster and stronger baseline for vision language model. arXiv preprint arXiv:2402.03766 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    Advances in Neural Information Processing Systems36 (2024)

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient fine- tuning of quantized llms. Advances in Neural Information Processing Systems36 (2024)

    work page 2024

  10. [10]

    VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

    Dong, M., Cai, H., Li, J., Zhou, S., Ren, B., Peng, K., Fu, Y.: Visnec: Measuring and leveraging visual necessity for multimodal instruction tuning. arXiv preprint arXiv:2603.01195 (2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2017)

    Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., Srebro, N.: Im- plicit regularization in matrix factorization. In: Advances in Neural Information Processing Systems (NeurIPS) (2017)

    work page 2017

  12. [12]

    Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    Han, Z., Gao, C., Liu, J., Zhang, S.Q., et al.: Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    LoRA+: Efficient Low Rank Adaptation of Large Models

    Hayou, S., Ghosh, N., Yu, B.: Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    In: International Conference on Learning Representations Dive into the implicit biases of low-rank vision-language alignment 17

    Hu, E.J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. In: International Conference on Learning Representations Dive into the implicit biases of low-rank vision-language alignment 17

    work page

  15. [15]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    work page 2019

  16. [16]

    HuggAN: WikiArt Dataset.https://huggingface.co/datasets/huggan/wikiart (2022), accessed: 2024-12-19

    work page 2022

  17. [17]

    In: The Twelfth International Confer- ence on Learning Representations (2022)

    Hyeon-Woo, N., Ye-Bin, M., Oh, T.H.: Fedpara: Low-rank hadamard product for communication-efficient federated learning. In: The Twelfth International Confer- ence on Learning Representations (2022)

    work page 2022

  18. [18]

    In: International Conference on Machine Learning (ICML) (2024)

    Jang, U., Lee, J.D., Ryu, E.K.: LoRA training in the NTK regime has no spurious local minima. In: International Conference on Machine Learning (ICML) (2024)

    work page 2024

  19. [19]

    In: Proc

    Khosla, A., Jayadevaprakash, N., Yao, B., Li, F.F.: Novel dataset for fine-grained image categorization: Stanford dogs. In: Proc. CVPR workshop on fine-grained visual categorization (FGVC). vol. 2 (2011)

    work page 2011

  20. [20]

    International journal of computer vision123(1), 32–73 (2017)

    Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision123(1), 32–73 (2017)

    work page 2017

  21. [21]

    Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. ICML pp. 19730– 19742 (2023)

    work page 2023

  22. [22]

    Evaluating Object Hallucination in Large Vision-Language Models

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models

    Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W., Zhao, T.: Loftq: Lora-fine-tuning-aware quantization for large language models. arXiv preprint arXiv:2310.08659 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  24. [24]

    In: European conference on computer vision

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)

    work page 2014

  25. [25]

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning (2023)

    work page 2023

  26. [26]

    NeurIPS36(2024)

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. NeurIPS36(2024)

    work page 2024

  27. [27]

    DoRA: Weight-Decomposed Low-Rank Adaptation

    Liu, S.Y., Wang, C.Y., Yin, H., Molchanov, P., Wang, Y.C.F., Cheng, K.T., Chen, M.H.: Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al.: Mmbench: Is your multi-modal model an all-around player? In: European conference on computer vision. pp. 216–233. Springer (2024)

    work page 2024

  29. [29]

    In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)

    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)

    work page 2015

  30. [30]

    Advances in Neural Information Processing Systems 35, 2507–2521 (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507–2521 (2022)

    work page 2022

  31. [31]

    IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

    Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., Zhu, S.C.: Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.13214 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  32. [32]

    Advances in Neural Information Processing Systems (2024) 18 M

    Ma, F., Xue, H., Zhou, Y., Wang, G., Rao, F., Yan, S., Zhang, Y., Wu, S., Shou, M.Z., Sun, X.: Visual perception by large language model’s weights. Advances in Neural Information Processing Systems (2024) 18 M. Shi et al

    work page 2024

  33. [33]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    Meng, F., Wang, Z., Zhang, M.: PiSSA: Principal singular values and singular vec- tors adaptation of large language models. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    work page 2024

  34. [34]

    In: 2019 international conference on document analysis and recognition (ICDAR)

    Mishra, A., Shekhar, S., Singh, A.K., Chakraborty, A.: Ocr-vqa: Visual question answering by reading text in images. In: 2019 international conference on document analysis and recognition (ICDAR). pp. 947–952. IEEE (2019)

    work page 2019

  35. [35]

    Sampling Theory, Signal Processing, and Data Analysis20(2), 11 (2022)

    Mixon, D.G., Parshall, H., Pi, J.: Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis20(2), 11 (2022)

    work page 2022

  36. [36]

    DINOv2: Learning Robust Visual Features without Supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  37. [37]

    Advances in neural information processing systems 24(2011)

    Ordonez, V., Kulkarni, G., Berg, T.: Im2text: Describing images using 1 mil- lion captioned photographs. Advances in neural information processing systems 24(2011)

    work page 2011

  38. [38]

    In: ICML

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)

    work page 2021

  39. [39]

    SAM 2: Segment Anything in Images and Videos

    Ravi, N., Gabeur, V., Hu, Y.T., Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., et al.: Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    In: Advances in Neural Information Processing Systems (NeurIPS) (2020)

    Razin, N., Cohen, N.: Implicit regularization in deep learning may not be explain- able by norms. In: Advances in Neural Information Processing Systems (NeurIPS) (2020)

    work page 2020

  41. [41]

    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge (2015),https://arxiv.org/abs/1409.0575

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  42. [42]

    Advances in neural information processing systems35, 25278–25294 (2022)

    Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al.: Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in neural information processing systems35, 25278–25294 (2022)

    work page 2022

  43. [43]

    In: Proceed- ings of ACL (2018)

    Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proceed- ings of ACL (2018)

    work page 2018

  44. [44]

    In: CVPR

    Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards VQA models that can read. In: CVPR. pp. 8317–8326 (2019)

    work page 2019

  45. [45]

    Team, K.: Kimi k2.5: Visual agentic intelligence.https://www.kimi.com/blog/ kimi-k2-5(2025)

    work page 2025

  46. [46]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., et al.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, lo- calization, and dense features. arXiv preprint arXiv:2502.14786 (2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [47]

    Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)

    work page 2011

  48. [48]

    QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models

    Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q.: Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  49. [49]

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, Dive into the implicit biases of low-rank vision-language alignment 19 J., Dang, K., Bao, K., Yang, K.,...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  50. [50]

    MiniCPM-V: A GPT-4V Level MLLM on Your Phone

    Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al.: Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  51. [51]

    In: The Twelfth International Conference on Learning Representations (2023)

    Yeh, S.Y., Hsieh, Y.G., Gao, Z., Yang, B.B., Oh, G., Gong, Y.: Navigating text-to- image customization: From lycoris fine-tuning to model evaluation. In: The Twelfth International Conference on Learning Representations (2023)

    work page 2023

  52. [52]

    National Science Review11(12), nwae403 (2024)

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. National Science Review11(12), nwae403 (2024)

    work page 2024

  53. [53]

    MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities

    Yu, W., Yang, Z., Ren, L., Li, L., Wang, J., Lin, K., Lin, C.C., Liu, Z., Wang, L., Wang, X.: Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.00765 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  54. [54]

    Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yu, B., Zhang, G., Sun, H., et al.: Mmmu-pro: A more robust multi-discipline multimodal under- standingbenchmark.In:Proceedingsofthe63rdAnnualMeetingoftheAssociation for Computational Linguistics (Volume 1: Long Papers). pp. 15134–15186 (2025)

    work page 2025

  55. [55]

    arXiv preprint arXiv:2308.10792 (2023)

    Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., et al.: Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792 (2023)

    work page arXiv 2023

  56. [56]

    In: Findings of the Association for Computational Linguistics: ACL 2025

    Zhou, S., Wang, S., Yuan, Z., Shi, M., Shang, Y., Yang, D.: Gsq-tuning: Group- shared exponents integer in fully quantized training for llms on-device fine-tuning. In: Findings of the Association for Computational Linguistics: ACL 2025. pp. 22971–22988 (2025)

    work page 2025

  57. [57]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

    Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023) 20 M. Shi et al. Appendix 1 Implicit Biases: Low-Rank Alignment 1.1 Preliminary: First-Order and Symbols Due to the high-order tensor and Kronecker structures employed in our de...

    work page internal anchor Pith review Pith/arXiv arXiv 2023