Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LAVCap claims that optimal-transport alignment and fusion of audio and visual tokens outperforms prior audio captioning systems on AudioCaps without extra training data.

desk verdict Strong AudioCaps results and a clean ablation, but the OT alignment loss is permutation-invariant and cannot be doing what the paper claims. read the letter →

arxiv 2501.09291 v2 pith:UHXQFTPG submitted 2025-01-16 cs.MM cs.AIcs.SDeess.AS

Kyeongha Rho , Hyeongkeun Lee , Valentio Iverson , Joon Son Chung This is my paper · ORCID
classification cs.MMcs.AIcs.SDeess.AS
keywords audiocaptioningaudio-visualfusionoptimaltransportSinkhornalgorithmlargelanguagemodelsLow-RankAdaptationCapsmultimodalalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LAVCap, a captioning system that describes a 10-second audio clip by also looking at the video frames that accompany it. The authors' central claim is that the usual failure of audio-visual captioning models is a modality gap: audio and visual features live in different spaces, so naively concatenating them or using cross-attention wastes the visual information. They propose treating the alignment of audio and visual tokens as an optimal transport problem, computing a soft assignment matrix between the two token sets, and using that matrix both as a training loss and as the attention weights for fusion. With a frozen CLIP visual encoder and LoRA-adapted audio encoder and Llama 2 decoder, LAVCap reports SPIDEr 51.7 and CIDEr 84.9 on the AudioCaps test set, the highest among the models compared, despite not pre-training on extra captioning datasets. The authors also report a user study in which their generated captions receive higher mean opinion scores than the ground-truth audio-only captions.

What carries the argument

The central object is the optimal transport assignment map $Q^*\in\mathbb{R}^{N_a\times N_v}$, obtained by solving a regularized transport problem over the dot-product similarity matrix $S$ between audio and visual tokens with Sinkhorn-Knopp iterations. $Q^*$ is doubly stochastic, so each audio token is distributed over visual tokens and vice versa. The paper uses $Q^*$ twice: in $L_{OT}$ (a cross-modal contrastive loss over the row/column pairs of $Q^*$ and $S$) to pull corresponding tokens together, and in OT-Att to re-weight the other modality's features by $\hat{h}_a = h_a + Q^* h_v$ and symmetrically for $\hat{h}_v$, with the two attended sequences concatenated and projected into the LLM's latent space. Because $Q^*$ is computed by a fixed algorithm from $S$, the fusion module adds no learnable parameters, which the authors argue is a data-efficient way to fuse modalities when training data is scarce.

What would settle it

Replace the learned similarity matrix $S$ in Eqs. (2)-(5) with a random matrix (or an identity-like constant) while keeping the Sinkhorn iterations, losses, and all other training choices unchanged; if the resulting SPIDEr stays close to the reported 51.7 rather than collapsing toward the no-OT baseline of 48.5, the transport plan itself is not carrying the claimed alignment and the gain must come from elsewhere in the pipeline.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the optimal transport assignment map $Q^*$ between audio tokens and visual tokens is a dual-purpose object: when its row/column similarities are pushed to be large within a sample under $L_{OT}$, it coaxes a frozen CLIP visual encoder and a trainable audio encoder into a shared semantic geometry, and when $Q^*$ is used directly as attention weights in the OT-Att fusion, it lets each modality attend to the other without any learnable fusion parameters. The authors show by ablation that adding visual features without $L_{OT}$ barely helps (SPIDEr 47.6 to 48.5), while adding $L_{OT}$ brings the full model to 51.0 on the validation-visible ablation, and the final architecture reaches 51.7 on the test split. The method therefore claims to be a data-efficient recipe: the OT map replaces a learned fusion layer, and LoRA on the audio encoder and LLM decoder substitute for full fine-tuning.

Load-bearing premise

The load-bearing premise is that the similarity matrix $S$ between frozen CLIP visual tokens and trainable CED audio tokens meaningfully reflects semantic correspondence, because the same $S$ that produces the transport plan is also the target of the alignment loss; if the two token spaces are not comparable at the token level, the transport plan is arbitrary and the OT loss would push the audio encoder toward meaningless structure.

Editorial extensions

If this is right

  • LAVCap's reported SPIDEr 51.7 and CIDEr 84.9 on the AudioCaps test split would be the best numbers among the models compared in Table I, including systems pre-trained on additional captioning datasets.
  • The OT loss plus OT-Att combination would make audio-visual captioning practical in data-scarce settings, since the only trainable components are LoRA adapters and a linear projector.
  • The OT assignment map doubles as an attention matrix, eliminating the need for a learned fusion module like a Q-Former or cross-attention in this pipeline.
  • Because the visual encoder is frozen, the framework suggests that any frozen visual backbone can be plugged in and aligned to audio through the same OT procedure.
  • The user study result, if reliable, implies that audio-visual captions can be judged as more accurate than audio-only ground truth, which would argue for collecting video-grounded captions in future datasets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would apply the same dual-use OT map to video-text or image-text captioning, where token-level correspondences are also loose and paired data is limited.
  • If the OT map is truly capturing semantic correspondences, its entries should be interpretable as attention weights that highlight the visual tokens (e.g., objects or scene regions) responsible for a given audio event; visualizing these maps on the AudioCaps test clips would make the claimed alignment concrete.
  • The paper's success with a frozen visual encoder and LoRA-tuned audio encoder suggests the OT loss may be doing the heavy lifting of cross-modal alignment, which would imply that even weaker audio encoders could be aligned to CLIP visual features by this loss alone.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. LAVCap is an audio-visual captioning system that combines a frozen CLIP ViT-L/14 visual encoder, a CED audio encoder adapted with LoRA, and a Llama-2 7B decoder adapted with LoRA. Its two novel components are an optimal-transport (OT) alignment loss, Eq. (4), and an OT-attention fusion module, Eq. (5), both built from the Sinkhorn assignment Q* of the cross-modal similarity matrix S. Training uses a weighted combination of captioning cross-entropy and OT loss, Eq. (9). On AudioCaps the method reports SPIDEr 51.7, CIDEr 84.9, and METEOR 26.2 without additional caption datasets or post-processing, and ablations show each component contributes.

Significance. If the reported numbers are reproducible, LAVCap is a data-efficient state-of-the-art result for audio captioning with visual side information, and the parameter-free OT-attention fusion is a simple and interesting design choice. The paper has notable strengths: all components are ablated under the same backbone (Tables II-IV), the code is publicly released, and the MOS user study includes confidence intervals. The central mechanistic claim, however, is not yet established: the OT loss in Eq. (4) is permutation-invariant and therefore cannot by itself select a semantically correct audio-visual token correspondence. The contribution would be convincing if the authors provided a shuffled-token control, an analysis of Q*, or a comparison with a standard audio-visual contrastive alignment loss. The work is suitable for a major revision rather than rejection because the empirical result and the fusion module remain plausible and the missing analyses are feasible.

major comments (3)
  1. [Section II-B, Eq. (4)] The claim that L_OT 'bridges the modality gap' is not supported, because the loss is invariant under arbitrary permutations of the audio or visual tokens. For a permutation matrix P, replacing S by S P changes the Sinkhorn solution to Q* P (the marginals are uniform), so each inner product in Eq. (4) is either unchanged or relabeled, and the loss value is identical; the same holds for row permutations of S. Consequently L_OT cannot favor one token correspondence over another; it only encourages rows and columns of Q* to be self-consistent with rows and columns of S. The 2.5 SPIDEr gain from adding L_OT in Table II may therefore be a discriminative or regularization effect rather than semantic cross-modal alignment. Please add a shuffled-token or permutation-invariance control, an analysis of the learned Q* (visualization or correspondence accuracy), or a comparison against a standard audio-visual InfoNCE loss trained on the same features.
  2. [Section III-B, Table I] The headline result rests on a single evaluation run. Several advantages over the strongest prior systems are small (SPIDEr 51.7 vs. 50.7 for AutoCap; METEOR 26.2 vs. 26.7 for LOAE), and no confidence intervals, multiple seeds, or significance tests are reported for the automatic metrics. Please report mean +/- standard deviation over at least three seeds for the final model and the closest baselines, or a paired test over test captions, before claiming state-of-the-art performance.
  3. [Section III-A and Eq. (9)] The paper omits several quantities needed to reproduce the objective and the experiments: the weights lambda_CE and lambda_OT, the temperature tau, the entropy regularization epsilon, the number of Sinkhorn iterations, the LoRA rank and alpha, and the batch size. It also does not state how the 'acquired' 48,595 training and 944 test clips were selected or whether prior methods in Table I were evaluated on the same subset. Please report these hyperparameters and the dataset subsetting procedure.
minor comments (6)
  1. [Throughout] The method name appears as 'LAVCap' in the title and abstract but as 'LA VCap' in the body; please standardize the spelling.
  2. [Eq. (2)] The same symbol Q is used for both the optimization variable and the feasible set; please use a distinct symbol, for example \mathcal{Q}, for the set of admissible transport plans.
  3. [Section II-B] The sentence 'OT loss encourages maximizing the similarity between audio-visual token sequences within a sample' does not match Eq. (4), which contrasts rows and columns of Q* with rows and columns of S; please rephrase to describe the actual objective.
  4. [Section III-A] The spectrogram size 1024 x 64 is inconsistent with a 10-second signal at 16 kHz and a 10-ms hop, which yields roughly 1,000 STFT frames; please clarify whether a mel filterbank or a resize is applied and report the exact input format.
  5. [Table I] Many entries for baseline methods are dashes; please state whether those metrics were not reported in the original papers or are omitted here.
  6. [Section III-A] The decoding hyperparameters for Llama 2 at inference (e.g., temperature, beam size, sampling strategy) are not reported; since the paper disclaims post-processing, these settings matter for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LAVCap's optimal transport loss is an auxiliary training regularizer, the final captioning predictions are scored against human annotations, and no load-bearing self-citation is used.

full rationale

The core training objective (Eq. 9) combines a cross-entropy captioning loss L_CE (Eq. 8) with an OT regularizer L_OT (Eq. 4). L_OT is computed from the similarity matrix S and the Sinkhorn plan Q* derived from S (Eqs. 2-3), so it is self-referential as a regularizer, but this does not make the paper's predictions circular: the text tokens y_i are generated by the LLM and evaluated against human ground-truth captions with external metrics (SPIDEr, CIDEr, METEOR), not against any quantity derived from the model's own outputs. The OT components are trained or directly computed from input features, not fitted values substituted for the reported metrics. The method does not invoke a uniqueness theorem or a prior result by the same authors; all cited building blocks (Sinkhorn [18], CLIP [21], CED [20], Llama 2 [22]) are external and independently available. The reviewer's concern that L_OT is invariant under token permutations is a critique of whether the mechanism achieves semantic alignment, not evidence that the reported result is equivalent to its input by construction. The ablation tables provide empirical, externally falsifiable support for the contribution. Therefore no specific circular step can be quoted, and the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities or forces. The free parameters are all hyperparameters of the OT and contrastive losses that are not reported, which is the main reproducibility gap. The axioms are standard domain assumptions about cross-modal alignment and LLM token comprehension, with the first being the most fragile.

free parameters (6)
  • lambda_OT (weight of OT loss) = not reported
    The final objective is lambda_CE L_CE + lambda_OT L_OT (Eq. 9). The value is not given, yet it controls the contribution of the alignment loss to training.
  • tau (temperature in OT loss) = not reported
    Appears in Eq. (4) as the temperature in the contrastive exponentials. Not specified.
  • epsilon (entropy regularization) = not reported
    Controls smoothness of the optimal transport plan in Eqs. (2) and (3). Not specified.
  • Number of Sinkhorn iterations = not reported
    Needed to compute Q* in Eq. (3), but not given.
  • LoRA rank = not reported
    LoRA is used for both encoder and decoder (Table IV), but the rank is not disclosed.
  • Batch size = not reported
    Not mentioned in the training details, affects the contrastive learning and OT optimization.
assumptions (3)
  • domain assumption Audio and visual tokens can be meaningfully assigned to each other via an optimal transport plan computed from cosine similarities.
    Eqs. (2)-(5) assume that the OT plan Q between CED audio features and CLIP visual features reflects semantic correspondence. The ablation in Table II shows that without the OT loss, adding visual features gives little benefit, indicating the entire performance gain rests on this alignment being learnable.
  • domain assumption Frozen CLIP ViT-L/14 and CED encoders provide features in a comparable vector space such that dot products in S are meaningful.
    The similarity matrix S in Eq. (2) is computed directly from the two encoder outputs; no learned projection is applied before computing S (only a linear projector after fusion, Eq. (6)). The encoders are trained on different tasks (image-text and audio tagging), so the assumption of comparability is load-bearing and untested in the paper.
  • domain assumption The LLM (Llama 2 7B) can interpret concatenated audio-visual tokens and caption them appropriately.
    The method relies on the LLM decoder understanding the fused audio-visual token sequence (Section II-D). This is a general assumption in LLM-based multimodal systems and is not formally justified, but it is consistent with prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport." pith.science (2026). https://pith.science/paper/UHXQFTPG

@misc{pith2026250109291,
  author       = {Pith},
  title        = {Pith review of: LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHXQFTPG}},
  note         = {Machine review of arXiv:2501.09291}
}
read the original abstract

Automated audio captioning is a task that generates textual descriptions for audio content, and recent studies have explored using visual information to enhance captioning quality. However, current methods often fail to effectively fuse audio and visual data, missing important semantic cues from each modality. To address this, we introduce LAVCap, a large language model (LLM)-based audio-visual captioning framework that effectively integrates visual information with audio to improve audio captioning performance. LAVCap employs an optimal transport-based alignment loss to bridge the modality gap between audio and visual features, enabling more effective semantic extraction. Additionally, we propose an optimal transport attention module that enhances audio-visual fusion using an optimal transport assignment map. Combined with the optimal training strategy, experimental results demonstrate that each component of our framework is effective. LAVCap outperforms existing state-of-the-art methods on the AudioCaps dataset, without relying on large datasets or post-processing. Code is available at https://github.com/NAVER-INTEL-Co-Lab/gaudi-lavcap.

Figures

Figures reproduced from arXiv: 2501.09291 by the authors.

Figure 1
Figure 1. (a) Overview of the proposed LAVCap Framework. (b) Detail of the Optimal Transport Fusion module. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Qualitative results of captions generated from models trained solely [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Optimal Transport-based Semantic Alignment for LLM-based Audio-Visual Speech Recognition

    cs.SD 2026-07 conditional novelty 5.0 of 10

    OT couplings that map Whisper and AV-HuBERT features onto LLaMA token embeddings, used as soft contrastive labels, yield SOTA LRS3-TED AVSR under clean and noisy SNRs.

  2. Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning

    cs.MM 2025-05 conditional novelty 5.0 of 10

    Entropy-aware gating and shuffled audio-video training pairs improve robustness to audiovisual mismatch in video-guided audio captioning.

  3. Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model

    cs.MM 2025-05 conditional novelty 5.0 of 10

    A chain-of-thought fine-tuned vision-language model can infer audio descriptions from silent videos, and using those descriptions as prompts improves video-to-audio generation.

  4. WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction

    cs.SD 2025-06 conditional novelty 4.0 of 10

    WhisQ uses Whisper and Qwen with co-attention and optimal transport to predict music quality and text-alignment scores, but its reported improvements do not match its own data.

Reference graph

Works this paper leans on

29 extracted references · 25 canonical work pages · cited by 4 Pith papers

  1. [1]

    Per- sonalized dialogue generation with persona-adaptive attention,

    Q. Huang, Y . Zhang, T. Ko, X. Liu, B. Wu, W. Wang, and H. Tang, “Per- sonalized dialogue generation with persona-adaptive attention,” in Pro- ceedings of the AAAI Conference on Artificial Intelligence , pp. 12916– 12923, 2023

    work page 2023

  2. [2]

    Audio captioning transformer,

    X. Mei, X. Liu, Q. Huang, M. D. Plumbley, and W. Wang, “Audio captioning transformer,” in Proceedings of the 6th Detection and Clas- sification of Acoustic Scenes and Events 2021 Workshop , pp. 211–215, 2021

    work page 2021

  3. [3]

    Automated audio captioning by fine-tuning bart with audioset tags,

    F. Gontier, R. Serizel, and C. Cerisara, “Automated audio captioning by fine-tuning bart with audioset tags,” in DCASE 2021 - 6th Workshop on Detection and Classification of Acoustic Scenes and Events , 2021

    work page 2021

  4. [4]

    Prefix tuning for automated audio captioning,

    M. Kim, K. Sung-Bin, and T.-H. Oh, “Prefix tuning for automated audio captioning,” in International Conference on Acoustics, Speech and Signal Processing, pp. 1–5, 2023

    work page 2023

  5. [5]

    EnCLAP: Combining neural audio codec and audio-text joint embedding for automated audio cap- tioning,

    J. Kim, J. Jung, J. Lee, and S. H. Woo, “EnCLAP: Combining neural audio codec and audio-text joint embedding for automated audio cap- tioning,” in International Conference on Acoustics, Speech and Signal Processing, pp. 6735–6739, 2024

    work page 2024

  6. [6]

    WavCaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumbley, Y . Zou, and W. Wang, “WavCaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    work page 2024

  7. [7]

    CoNeTTE: An efficient audio captioning system leveraging multiple datasets with task embedding,

    E. Labb, T. Pellegrini, J. Pinquier, et al., “CoNeTTE: An efficient audio captioning system leveraging multiple datasets with task embedding,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    work page 2024

  8. [8]

    Enhancing automated audio captioning via large language models with optimized audio encoding,

    J. Liu, G. Li, J. Zhang, H. Dinkel, Y . Wang, Z. Yan, Y . Wang, and B. Wang, “Enhancing automated audio captioning via large language models with optimized audio encoding,” in Interspeech, pp. 1135–1139, 2024

    work page 2024

Show all 29 references
  1. [9]

    Taming data and transformers for audio generation,

    M. Haji-Ali, W. Menapace, A. Siarohin, G. Balakrishnan, S. Tulyakov, and V . Ordonez, “Taming data and transformers for audio generation,” arXiv preprint arXiv:2406.19388 , 2024

    2024 arXiv

  2. [10]

    PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “PANNs: Large-scale pretrained audio neural networks for audio pattern recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  3. [11]

    HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,

    K. Chen, X. Du, B. Zhu, Z. Ma, T. Berg-Kirkpatrick, and S. Dubnov, “HTS-AT: A hierarchical token-semantic audio transformer for sound classification and detection,” in International Conference on Acoustics, Speech and Signal Processing , pp. 646–650, 2022

    2022

  4. [12]

    BEATs: audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei, “BEATs: audio pre-training with acoustic tokenizers,” in Proceedings of the 40th International Conference on Machine Learning , 2023

    2023

  5. [13]

    CLAP: Learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “CLAP: Learning audio concepts from natural language supervision,” in International Conference on Acoustics, Speech and Signal Processing , pp. 1–5, 2023

    2023

  6. [14]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Transactions on Machine Learning Research, 2023

    2023

  7. [15]

    Visually-aware audio captioning with adaptive audio-visual attention,

    X. Liu, Q. Huang, X. Mei, H. Liu, Q. Kong, J. Sun, S. Li, T. Ko, Y . Zhang, L. H. Tang, et al. , “Visually-aware audio captioning with adaptive audio-visual attention,” in Interspeech, pp. 2838–842, 2023

    2023

  8. [16]

    A VCap: Leveraging audio-visual features as text tokens for captioning,

    J. Kim, J. Shin, and J. Kim, “A VCap: Leveraging audio-visual features as text tokens for captioning,” in Interspeech, 2024

    2024

  9. [17]

    Multi- granularity correspondence learning from long-term noisy videos,

    Y . Lin, J. Zhang, Z. Huang, J. Liu, Z. Wen, and X. Peng, “Multi- granularity correspondence learning from long-term noisy videos,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  10. [18]

    Sinkhorn distances: Lightspeed computation of optimal transport,

    M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” in Proceedings of the 26th International Conference on Neural Information Processing Systems , p. 2292–2300, 2013

    2013

  11. [19]

    Audiocaps: Generating captions for audios in the wild,

    C. D. Kim, B. Kim, H. Lee, and G. Kim, “Audiocaps: Generating captions for audios in the wild,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), ...

    2019

  12. [20]

    Ced: Consistent ensemble distillation for audio tagging,

    H. Dinkel, Y . Wang, Z. Yan, J. Zhang, and Y . Wang, “Ced: Consistent ensemble distillation for audio tagging,” in International Conference on Acoustics, Speech and Signal Processing , pp. 291–295, 2024

    2024

  13. [21]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in Proceedings of the 38th International Conference on Machine Learning , pp. 8748–8763, 2021

    2021

  14. [22]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  15. [23]

    LoRA: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-rank adaptation of large language models,” in The Tenth International Conference on Learning Representations , 2022

    2022

  16. [24]

    BLEU: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , p. 311–318, 2002

    2002

  17. [25]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out , pp. 74–81, 2004

    2004

  18. [26]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Pro- ceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pp. 65–72, 2005

    2005

  19. [27]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 4566– 4575, 2015

    2015

  20. [28]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in Proceedings of the European Conference on Computer Vision , pp. 382–398, 2016

    2016

  21. [29]

    Improved image captioning via policy gradient optimization of spider,

    S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Improved image captioning via policy gradient optimization of spider,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 873– 881, 2017

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.