Pith. sign in

REVIEW 5 major objections 4 minor 24 references

Small multimodal chain-of-thought reasoners can recover fine-grained visual-semantic differences by distilling an SVD steering vector derived from attention-masked images.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:03 UTC pith:KDJ2K6IS

load-bearing objection VSSD is a plausible and mostly well-executed combination of perturbation and steering-vector distillation with real gains on two benchmarks; the main missing piece is a comparison to the soft-negative-sampling baseline that the paper itself cites, plus standard rigor (multiple seeds, error bars). the 5 major comments →

arxiv 2607.22013 v1 pith:KDJ2K6IS submitted 2026-07-24 cs.CV cs.AI

Visual Saliency Steering Distillation for Multimodal Chain-of-Thought Reasoning

Hao Yang , Jin Wang , Xuejie Zhang This is my paper
classification cs.CV cs.AI
keywords multimodal chain-of-thought reasoningvisual saliencyperturbed imagesteering vectorsingular value decompositioninter-layer distillationfine-grained cross-modal fusionscience question answering
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal chain-of-thought models that fuse vision and text often collapse tiny cross-modal differences: once fused, a question paired with two similar images, or two questions paired with one image, becomes nearly indistinguishable, and small models suffer most. This paper's proposal, Visual Saliency Steering Distillation (VSSD), attacks exactly that failure. A teacher model's attention map identifies the image regions the question depends on; VSSD blanks those regions to create a perturbed image, then measures how the student decoder's hidden states shift between the original and perturbed inputs. The top-1 right singular vector of that per-layer difference is taken as the dominant steering direction and injected into the student's last decoder layers through a normalized distillation loss. If the paper's claims hold, a 223M-parameter model reaches 89.67% accuracy on ScienceQA, a 738M-parameter version reaches 93.40%, and the same recipe reaches 73.19% on M3CoT, with rationale quality improving alongside answer accuracy.

Core claim

On its own terms, the paper establishes that the confusion between same-text/close-image and same-image/close-text inputs in small multimodal CoT models can be reduced without extra data or a larger student. The discovery is that a counterfactual visual input — the original image with its attention-highlighted, question-relevant regions masked away — exposes a stable, low-dimensional directional signal in the decoder's hidden states. SVD of the layer-wise difference tensor E = H+ − H− yields a top-1 right singular vector; adding it to the mean-pooled original representation, normalized to the original norm, and enforcing a layer-wise distillation loss over the last L layers sharpens the mode

What carries the argument

Central object: the steering vector — the top-1 right singular vector of the difference between decoder hidden states for the original image and its attention-masked perturbed image. For each of the final L layers the paper forms E = H+ − H−, applies SVD to E, and keeps V_{:,1}; it then computes z+ = mean-pool over tokens of H+, and forms a normalized steered representation z~ = normalize(z+ + α·V_{:,1}) to the norm of z+. A layer-wise distillation loss drives z+ toward z~, added to the standard negative log-likelihood with weight β. The perturbed image is itself produced by a binary mask built from aggregated, upsampled, smoothed cross-attention maps of a multimodal large language model; ma

Load-bearing premise

The load-bearing assumption is that the top-1 right singular vector of the hidden-state difference between the original and masked image captures a task-relevant semantic shift, rather than arbitrary noise or low-level artifacts from image masking.

What would settle it

Run VSSD on ScienceQA with the attention-guided mask replaced by a random mask of matched sparsity, keeping every other step identical; if average accuracy does not drop clearly below 89.67% (Base) and 93.40% (Large), the saliency and SVD steering direction are not the cause of the reported gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • VSSD-Base (223M parameters) reaches 89.67% average accuracy on ScienceQA, and VSSD-Large (738M) reaches 93.40%, both above the paper's reported fine-tuned baselines.
  • On M3CoT, VSSD-Base reaches 73.19% average accuracy, while removing the inter-layer distillation component drops it to 61.57%.
  • Rationale generation benefits alongside answer accuracy: RougeL scores rise when the perturbed image and the steering distillation are active.
  • Ablations attribute the gain to both components: removing the perturbed image lowers ScienceQA accuracy by about 2.2 points; removing the inter-layer distillation lowers it by about 4.1 points.
  • Feature-similarity measurements show lower cosine similarity between same-text/close-image and same-image/close-text pairs at both fusion and decoder stages, directly reducing the confusion the paper targets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the steering vector is extracted from the student's own hidden states, VSSD may amplify the model's existing inductive biases instead of importing genuinely new visual knowledge; comparing it against a steering target labeled by an external visual-semantic supervision signal would clarify this.
  • Editorial inference: the masking threshold, top-k, and the fixed choice of the last L=2 decoder layers are likely benchmark-specific; adaptive selection of L or of the attention threshold could be tested across reasoning domains.
  • Editorial inference: the method needs only a trainable text decoder, a frozen vision encoder, and teacher attention maps, so the same recipe should transfer to other vision-language generation tasks, and one testable extension is whether a steering vector learned on one dataset transfers zero-shot to another.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes Visual Saliency Steering Distillation (VSSD) for small multimodal chain-of-thought (CoT) models. VSSD uses an MLLM's cross-attention maps to mask salient image regions and construct a perturbed image. The student T5 model encodes both the original and perturbed images; the per-sample difference of decoder hidden states is decomposed by SVD, and the top-1 right singular vector is used as a "steering vector" that is added to the mean-pooled hidden states of the last L layers, with an L2 loss distilling the steered representation. Experiments on ScienceQA and M3CoT report strong accuracy (89.67% for 223M and 93.40% for 738M on ScienceQA; 73.19% on M3CoT for the base model), with ablations over the perturbed image and inter-layer distillation. However, the closest prior baseline—soft negative sampling [7]—is not evaluated, and the steering vector is extracted from the student's own hidden states, making the "distillation" self-referential. Several presentation issues, including an unreadable Figure 3, further reduce confidence in the supporting evidence.

Significance. If the results hold, VSSD is a simple, parameter-free regularizer that could improve small multimodal CoT models. The paper reports transparent hyperparameters, a fixed seed, and a code link, which are strengths for reproducibility. The method's significance depends on whether the SVD-derived steering direction is genuinely more informative than existing contrastive/negative-sample objectives. Because the closest prior method [7] is omitted from all comparisons, and because no analysis of the steering vector's content is provided, the paper currently does not establish this. The self-referential nature of the steering target also means the contribution is more accurately described as a regularizer than as cross-model distillation.

major comments (5)
  1. [§1, §3.1, Tables 1–2, Fig. 2] The closest prior baseline is missing. The paper cites Zheng et al. [7] in §1 as prior work on "modeling fine-grained differences during the CoT generation process," and Figure 2 labels a "CKD" (contrastive knowledge distillation) component. Yet [7] is absent from the baseline lists in §3.1 and from Tables 1 and 2. Without this control, the reported gains over MM-CoT, DDCoT, and Enigma-COT could be attributed to generic contrastive/negative-sample training already published in [7] rather than to the SVD-based steering vector. The authors should add [7] to the comparisons and ablate against a soft-negative-sampling loss or a similar non-SVD contrastive objective.
  2. [§2.3, Eqs. (6)–(7)] The load-bearing assumption that the top-1 right singular vector of H+ − H− captures a task-relevant semantic shift is not tested. The difference is computed from the student model's own hidden states, so the MLLM supplies only the perturbation mask, not the steering direction. This makes the approach self-referential: any pre-existing visual insensitivity in the student is baked into the target, and the method may simply amplify the student's current inductive biases. The paper should provide evidence about the content of v_steering—for example, by comparing it against random SVD directions, directions from BKG or random masks, or projections onto task-related embeddings—and should reconsider whether "distillation" is an accurate description.
  3. [§3.4, Fig. 3] Figure 3 is not legible in the submitted manuscript: the plots are replaced by raw PDF glyph tokens (strings of "/uni..."). As a result, the layer-selection results (L=2 optimal) and the cosine-similarity reductions cannot be inspected. This is the key evidence for §3.4. The authors must provide a properly rendered figure or, failing that, a table of the layer sweep and cosine-similarity values.
  4. [§3.2, Table 2; §3.1 Implementation Details] The paper uses a fixed seed (42) and reports no error bars or significance tests. Given that the M3CoT total drops from 73.19 to 61.57 when distillation is removed, variance across seeds is needed to assess the stability of the gains. In addition, the reported totals do not match the unweighted mean of the six per-domain columns (VSSD-Base: 73.19 vs. approximately 72.98; w/o distillation: 61.57 vs. approximately 62.77). The weighting over domains should be specified so the totals are verifiable.
  5. [§3.3, Table 3; §3.5, Table 4] The BKG baseline in Table 3 is a black image, which removes all visual content. This does not establish that the saliency-guided mask specifically is beneficial; any nontrivial perturbation might suffice. A comparison with random masking or attention-inverted masking is needed. Relatedly, the "w/o PI" ablation in Table 4 is ambiguous: if PI is removed, how is the difference H+ − H− computed? If the perturbed image is removed entirely, the ablation removes both the perturbation and the distillation at once, making the decomposition unclear.
minor comments (4)
  1. [Title page] The running title reads "Visual Saliency Steering Distilla TION" — a typo.
  2. [Table 1] Several numeric entries are concatenated without spaces (e.g., "95.9182.00", "91.5696.74", "88.9993.52"), making the table hard to read. Please format the values consistently.
  3. [Algorithm 1] The symbol A is used both for the predicted answer and for the attention map. Rename one of them (e.g., answer as a_pred, attention map as A_map) to avoid confusion.
  4. [§2.2] The sentence "Perturbed images are constructed guided by cross-attention maps from MLLMs" is grammatically awkward. Consider revising for clarity.

Circularity Check

1 steps flagged

Steering vectors are computed from the student's own hidden states (Eqs. 6-7), so the distillation target is a function of the model being trained; the MLLM only supplies the mask. External benchmark results remain independent.

specific steps
  1. self definitional [Section 2.3, Eqs. (6)-(9)]
    "E_l = H+_l − H−_l, E^(b)_l = UΣV^⊤ ... Select the top-1 right singular vector as the principal editing direction: v^(b)_steering = V_:,1, z+_l = MeanPool(H+_l). ... \hat z_l = z+_l + α·v_steering, \tilde z_l = \hat z_l · ||z+_l||_2 / (||\hat z_l||_2 + ε). ... L_distill = 1/L Σ_l || z+_l − \tilde z_l ||^2_2, L = L_NLL + β·L_distill."

    The steering vector v_steering is the top-1 right singular vector of the difference between the student's own hidden states for original and perturbed images (H+_l − H−_l). The distillation target \tilde z_l is then constructed from that same student hidden state z+_l plus α·v_steering, renormalized to the norm of z+_l. The loss minimizes the distance between z+_l and a function of z+_l and v_steering, i.e., a self-referential target. The MLLM contributes only the perturbation mask (Algorithm 1); no teacher hidden state or gradient enters the steering direction. Thus the 'distillation' cannot transfer MLLM knowledge beyond what the student already encodes; it is a self-distillation/contrastive regularizer. The reported accuracy on external benchmarks is not forced, so this is partial circu

full rationale

The central empirical claims (ScienceQA, M3CoT accuracies) are tested against held-out external benchmarks, so they are not circular in the usual prediction sense. The main circularity concern is internal: the steering vector used to define the distillation target is extracted from the very model being trained, making the target a rescaling of the student's own representation. This does not invalidate the empirical gains, but it weakens the paper's framing as distilling MLLM knowledge. The absence of reference [7] (soft negative sampling) from the baseline tables is a missing-control/novelty concern, not a circularity step: the paper does not invoke [7] as a load-bearing justification for VSSD's derivation. No uniqueness theorem or ansatz is imported from the authors' prior work. Overall, partial self-referentiality in the training objective with independent empirical content yields a score of 4.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 1 invented entities

The method rests on several hand-set hyperparameters (α, β, L) and unstated ones (τ, top-k, blur), plus the assumptions that MLLM attention is a reliable saliency oracle and that the dominant SVD direction of a hidden-state difference is semantically meaningful. No new physical entities are introduced; the steering vector is a mathematical artifact of the training procedure.

free parameters (7)
  • α (enhancement/scaling factor) = 0.1
    Controls mask normalization in Algorithm 1 and steering injection strength in Eq. (8); reported in Section 3.1 as chosen by hand.
  • β (distillation loss weight) = 0.2
    Balances NLL and L_distill in Eq. (9); reported as a fixed hyperparameter.
  • L (number of final decoder layers) = 2
    Selected as best via ablation in Fig. 3(a); directly controls which hidden states contribute to steering.
  • top-k (mask selection)
    Algorithm 1 uses top-k selection but k is not reported; affects which image regions are masked.
  • τ (attention mask threshold)
    Algorithm 1 thresholds normalized attention; value not reported.
  • Gaussian blur kernel/σ
    Used in Algorithm 1 before mask generation; parameters not specified.
  • Number of SVD components retained (top-1) = 1
    Eq. (7) keeps only the first right singular vector; a design choice that discards other directions.
axioms (4)
  • domain assumption MLLM cross-attention maps localize the image regions relevant to answering the question
    Algorithm 1 uses Qwen2.5-VL's attention to decide what to mask; if this localization is wrong, the perturbed image is not a meaningful counterfactual.
  • domain assumption The top-1 right singular vector of H+ − H− captures a task-relevant semantic direction
    Eqs. (6)-(7) assume the dominant difference between original and perturbed hidden states is semantically meaningful rather than dominated by low-level image variation.
  • ad hoc to paper The L2 distillation toward z+ + α·v_steering improves sensitivity without destroying the original representation
    Eqs. (8)-(9) define this as the training objective; it is a design choice, not derived from prior theory.
  • domain assumption T5 with a frozen DETR visual encoder and gated fusion is an adequate substrate for multimodal CoT
    Section 2.1 builds the method on this architecture; results may not transfer to other fusion schemes.
invented entities (1)
  • Steering vector (v_steering) no independent evidence
    purpose: A direction in decoder hidden-state space, extracted as the top-1 right singular vector of the difference between original and perturbed representations, used to shift hidden states during training.
    It is a construct derived from the model's own activations; the only evidence for its usefulness is the paper's ablation, not an independent falsifiable prediction.

pith-pipeline@v1.3.0-alltime-deepseek · 9458 in / 15569 out tokens · 146216 ms · 2026-08-01T06:03:09.252902+00:00 · methodology

0 comments
read the original abstract

Multimodal chain-of-thought (CoT) reasoning integrates visual and textual cues through step-by-step inference. In small models with limited token budgets, modality-interaction fusion often suppresses tiny cross-modal differences. In particular, multimodal CoT often struggles when different images pair with identical text or different texts pair with an identical image, making such inputs nearly indistinguishable after fusion. This study proposes Visual Saliency Steering Distillation (VSSD). VSSD leverages the attention maps of multimodal large language models to generate perturbed images that capture task-sensitive feature directions, and then applies singular value decomposition to extract dominant steering vectors to guide inter-layer distillation. Experiments on ScienceQA and M$^3$CoT demonstrate that VSSD improves rationale generation and answer inference. The code is available at https://github.com/BGWH123/VSSD.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 7 linked inside Pith

  1. [1]

    A natural approach is to leverage multimodal large language models (MLLMs) [1, 2] to enable CoT

    INTRODUCTION Multimodal chain-of-thought (CoT) captures visual-linguistic cues through explicit reasoning steps. A natural approach is to leverage multimodal large language models (MLLMs) [1, 2] to enable CoT. However, the large parameter size and high computational costs limit the deployment in resource- constrained scenarios. Therefore, research on mult...

    Pith/arXiv arXiv 2026

  2. [2]

    VISUAL SALIENCY STEERING DISTILLA TION The proposed VSSD has two components in Fig. 2: (i) gener- ating perturbed image to emphasize key visual cues, and (ii) extracting the main semantic shift between original and per- turbation features to guide inter-layer distillation. 2.1. Preliminaries Model Architecture. We adopt a two-stage scheme: ratio- nale gen...

    arXiv

  3. [3]

    Experiments Settings Dataset

    EXPERIMENT 3.1. Experiments Settings Dataset. The proposed VSSD was evaluated on the Sci- enceQA [9] benchmark, a multimodal chain-of-thought dataset with over 21,000 multiple-choice questions across three science subjects. Further tests were conducted on M3CoT [10], a more challenging variant of ScienceQA where each sample is paired with an image. Implem...

  4. [4]

    Existing methods often blur tiny cross-modal cues

    CONCLUSION This study proposes a visual saliency steering distillation (VSSD) to address the loss of fine-grained visual-semantic differences in small multimodal CoT methods. Existing methods often blur tiny cross-modal cues. VSSD generates attention-guided perturbed image to capture task-sensitive feature directions and distills steering vectors into int...

  5. [5]

    Hyperparametersαandβwere 0.1 and 0.2

    The input lengths were 512 (rationale) and 64 (answer). Hyperparametersαandβwere 0.1 and 0.2. The layerLis 2. A fixed seed 42 ensured reproducibility. Baselines.We compare our model with five categories: (1) Instruction-tuned LLMs: GPT-3.5, CoT-enhanced variants, ChatGPT, GPT-4 [9]; (2)Tool-augmented LLMs: Chameleon [12], VisualChatGPT [13], IdealGPT [14]...

    arXiv

  6. [6]

    Joint multimodal entity-relation extraction based on edge- enhanced graph alignment network and word-pair rela- tion tagging,

    Li Yuan, Yi Cai, Jin Wang, and Qing Li, “Joint multimodal entity-relation extraction based on edge- enhanced graph alignment network and word-pair rela- tion tagging,” inProceedings of the AAAI Conference on Artificial Intelligence, 2023, pp. 11051–11059

    2023

  7. [7]

    61966038 and 62266051, and the Postgraduate Research and Innovation Foundation of Yunnan University under Grant No.KC-252513133

    ACKNOWLEDGEMENT This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant Nos. 61966038 and 62266051, and the Postgraduate Research and Innovation Foundation of Yunnan University under Grant No.KC-252513133. The authors would like to thank the anonymous reviewers for their constructive comments

  8. [8]

    Gpt-4o system card,

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Ak- ila Welihinda, Alan Hayes, Alec Radford, et al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Visual instruction tuning,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, “Visual instruction tuning,”Advances in neu- ral information processing systems, vol. 36, pp. 34892– 34916, 2023

    2023

  10. [10]

    validate the effectiveness of VSSD. Our key contributions are as follows: I.We propose VSSD to address the limitation of small multimodal CoT models in capturing fine-grained visual-semantic differences.II.VSSD leverages saliency-based perturbation and SVD-guided steer- ing vector injection to enhance cross-modal sensitivity.III. Experiments on ScienceQA ...

  11. [11]

    Multimodal chain- of-thought reasoning in language models,

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola, “Multimodal chain- of-thought reasoning in language models,”arXiv preprint arXiv:2302.00923, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Boosting the power of small multimodal reason- ing models to match larger models with self-consistency training,

    Cheng Tan, Jingxuan Wei, Zhangyang Gao, Linzhuang Sun, Siyuan Li, Ruifeng Guo, Bihui Yu, and Stan Z Li, “Boosting the power of small multimodal reason- ing models to match larger models with self-consistency training,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 305–322

    2024

  13. [13]

    Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation,

    Kuanghong Liu, Jin Wang, Kangjian He, Dan Xu, and Xuejie Zhang, “Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation,” inProceedings of the 39th AAAI Con- ference on Artificial Intelligence (AAAI-2025), 2025, vol. 39, pp. 18897–18905

    2025

  14. [14]

    Enhancing semantics in multimodal chain of thought via soft negative sampling,

    Guangmin Zheng, Jin Wang, Xiaobing Zhou, and Xue- jie Zhang, “Enhancing semantics in multimodal chain of thought via soft negative sampling,” inProceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 6059–6076

    2024

  15. [15]

    Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive frame- work,

    Jingxuan Wei, Cheng Tan, Zhangyang Gao, Linzhuang Sun, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li, “Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive frame- work,”Neural Computing and Applications, vol. 36, no. 33, pp. 20849–20861, 2024

    2024

  16. [16]

    Learn to explain: Multimodal rea- soning via thought chains for science question answer- ing,

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, “Learn to explain: Multimodal rea- soning via thought chains for science question answer- ing,” inThe 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  17. [17]

    M 3cot: A novel bench- mark for multi-domain multi-step multi-modal chain-of- thought,

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che, “M 3cot: A novel bench- mark for multi-domain multi-step multi-modal chain-of- thought,” inProc. of ACL, 2024

    2024

  18. [18]

    Unifiedqa: Crossing format boundaries with a single qa system,

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord, Peter Clark, and Hannaneh Ha- jishirzi, “Unifiedqa: Crossing format boundaries with a single qa system,”arXiv preprint arXiv:2005.00700, 2020

    Pith/arXiv arXiv 2005

  19. [19]

    Chameleon: Plug-and-play compositional reasoning with large language models,

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao, “Chameleon: Plug-and-play compositional reasoning with large language models,”Advances in Neural Information Processing Systems, vol. 36, pp. 43447–43478, 2023

    2023

  20. [20]

    Visual chat- gpt: Talking, drawing and editing with visual foundation models,

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan, “Visual chat- gpt: Talking, drawing and editing with visual foundation models,”arXiv preprint arXiv:2303.04671, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Idealgpt: Iteratively decomposing vision and language reasoning via large language mod- els,

    Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A Ayyubi, Kai-Wei Chang, and Shih-Fu Chang, “Idealgpt: Iteratively decomposing vision and language reasoning via large language mod- els,”arXiv preprint arXiv:2305.14985, 2023

    Pith/arXiv arXiv 2023

  22. [22]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention,

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,”arXiv preprint arXiv:2303.16199, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Cheap and quick: Ef- ficient vision-language instruction tuning for large lan- guage models,

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xi- aoshuai Sun, and Rongrong Ji, “Cheap and quick: Ef- ficient vision-language instruction tuning for large lan- guage models,”Advances in Neural Information Pro- cessing Systems, vol. 36, pp. 29615–29627, 2023

    2023

  24. [24]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language mod- els,

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang, “Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language mod- els,”Advances in Neural Information Processing Sys- tems, vol. 36, pp. 5168–5191, 2023

    2023