Pith. sign in

REVIEW 2 major objections 4 minor 70 references

Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MMSLT shows that MLLM-written descriptions of sign language components, fused with video features during pre-training, set a new state of the art for gloss-free sign language translation on PHOENIX14T and CSL-Daily.

desk verdict Novel and plausible method for gloss-free SLT via MLLM descriptions, but the headline SOTA numbers are compromised by test-set selection in the ablations; a dev-selected rerun is needed before trusting the +2.33/+5.36 claims. read the letter →

arxiv 2411.16789 v2 pith:RS6YF33Q submitted 2024-11-25 cs.CV cs.CL

Jungeun Kim , Hyeongwoo Jeon , Jongseong Bae , Ha Young Kim This is my paper · ORCID
classification cs.CVcs.CL
keywords signlanguagetranslationgloss-freemultimodallargemodelsvisual-languagepre-trainingdescriptionmapperPHOENIX14TCSL-Dailycontrastivealignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that gloss-free sign language translation improves when the model sees both the sign video and a text description of the sign components (hand shape, gaze, mouth) generated by an off-the-shelf multimodal large language model. To keep inference cheap, a small description mapper learns to predict the embeddings of those descriptions from visual features alone, so the MLLM is only needed during pre-training. On PHOENIX14T, the method reports a test BLEU-4 of 25.73, beating the previous gloss-free state of the art by +2.33, and on CSL-Daily it reports 21.11, a +5.36 gain. The central claim is that the two modalities complement each other and that aligning the fused representation with spoken sentences bridges the modality gap better than using vision alone.

What carries the argument

The central mechanism is the multimodal-language pre-training module (MMLP), which contains four components: a description mapper (a two-layer MLP) that predicts the BERT [CLS] embedding of an MLLM-generated sign description from ResNet18 visual features; a modality adapter consisting of 1D convolution, max-pooling, and an MLP that fuses visual and predicted description features; a 12-layer mBART encoder fine-tuned with LoRA that extracts long-term multimodal features; and a contrastive alignment loss (with a learnable temperature) that pulls pooled multimodal features toward pooled spoken-sentence features from a frozen mBART encoder. The MLLM itself is used only in the GSD-MLLM pre-processing step, where a prompt focused on hands and face is fed to LLaVA-OneVision 7B to generate per-frame descriptions.

What would settle it

Run MMSLT's full training pipeline, but at inference replace the mapper's predicted description embeddings with the true BERT embeddings of the MLLM descriptions (i.e., use the MLLM at inference). If BLEU-4 does not improve over the mapper version, or if replacing the predicted embeddings with random vectors of the same shape preserves the reported BLEU-4, then the contribution attributed to the semantic content of the descriptions would be called into question.

Watch

Extended reading notes

Core claim

The paper claims that MMSLT, a gloss-free sign language translation framework, outperforms prior gloss-free methods by leveraging multimodal large language models to generate detailed textual descriptions of sign language components from video frames. During pre-training, the description features are fused with visual features through a modality adapter and aligned with spoken sentences via contrastive learning, and a description mapper predicts the description embeddings from visual features so that the MLLM is not needed at inference. On the PHOENIX14T and CSL-Daily benchmarks, the approach achieves state-of-the-art gloss-free BLEU-4 scores of 25.73 and 21.11 respectively, outperforming SignLLM by +2.33 and +5.36, and also improving ROUGE scores, indicating better handling of long phrases and context.

Load-bearing premise

The two-layer description mapper is assumed to predict the BERT embeddings of the MLLM descriptions from visual features closely enough that the model at inference retains the translation benefit of the descriptions, even though the MLLM itself is no longer present.

Editorial extensions

If this is right

  • Gloss-free sign language translation can be improved without any gloss annotations by adding MLLM-generated text descriptions as a second input modality during pre-training.
  • Because the description mapper approximates the description embeddings at inference, the computational cost of the MLLM is avoided at deployment, making the approach practical for real-time use.
  • The method generalizes across two languages and datasets (German PHOENIX14T and Chinese CSL-Daily) and across multiple MLLMs, with most tested MLLMs outperforming the previous gloss-free state of the art.
  • The choice of prompt matters: a prompt that explicitly asks for motion and gestures of the person focused on hands and face yields better descriptions and higher translation quality than simpler or more generic prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the description mapper is trained to predict BERT embeddings of the MLLM descriptions from visual features, the benefit at deployment ultimately depends on how much of the description content is already recoverable from the visual features; if the mapper mostly recovers information already present in the ResNet features, the text modality may be adding less than the pre-training alignment it
  • A natural extension would be to distill the MLLM directly into the mapper using the full description texts (or their token embeddings) as supervision, rather than only the sentence-level [CLS] embedding, which could retain more fine-grained information.
  • The same recipe could transfer to other video-to-language tasks, such as instructional video captioning or action recognition, where an MLLM can provide auxiliary textual descriptions of fine-grained motion without requiring manual annotation.
  • The reported gains could be tested for robustness by evaluating on a low-resource sign language dataset or on signers unseen during training, where the diversity of MLLM descriptions might have a larger or smaller effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes MMSLT, a gloss-free sign language translation framework that uses an off-the-shelf multimodal large language model (MLLM) to generate textual descriptions of sign language components from video frames. These descriptions are encoded with a frozen BERT and fused with ResNet visual features via a modality adapter; the fused features are aligned with spoken sentences through contrastive learning and passed to an mBART decoder for translation. A description mapper (a two-layer MLP) is trained to predict the description embeddings from visual features so that the MLLM is not needed at inference. Experiments on PHOENIX14T and CSL-Daily report state-of-the-art gloss-free results, e.g., +2.33 BLEU-4 over SignLLM on PHOENIX14T test and +5.36 on CSL-Daily test. Ablations in Sec. 5.3 show that each component contributes to the final performance.

Significance. If the reported gains hold under a sound evaluation protocol, the paper makes a meaningful contribution: it is the first to show that off-the-shelf MLLM-generated descriptions can be leveraged, without extra gloss supervision, to improve gloss-free SLT, and it provides a computationally efficient inference scheme via the description mapper. The method is clearly described, the code is released, and the ablations (Tables 3–6) are informative. The main risk is that the central 'state-of-the-art' claim rests on model choices made using the test set itself, which can substantially inflate the reported margins. A dev-selected re-run is needed to establish the actual gain.

major comments (2)
  1. [Sec. 5.3, Tables 3–6]
  2. [Sec. 4.2, Eq. (1)–(2); Table 3, rows (5)–(6)]
minor comments (4)
  1. [Table 6]
  2. [Table 4]
  3. [Eq. (7) and Sec. 4.3]
  4. [Fig. 4 caption]

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-circularity: MMSLT's claimed gains rest on external benchmarks and off-the-shelf MLLMs; the test-set selection in Sec. 5.3 is a validity risk, not a circular step.

full rationale

MMSLT's derivation chain is self-contained against external benchmarks. The SL descriptions are produced by off-the-shelf MLLMs (e.g., LLaVA-OneVision), not by the authors' own prior results, and the description mapper is a supervised regressor (Eq. 1-2) trained to mimic BERT embeddings of those descriptions; at inference it replaces the MLLM, which is a standard distillation design rather than a fitted input being passed off as an independent prediction. The final translation scores in Tables 1-2 are measured on the held-out dev/test sets of PHOENIX14T and CSL-Daily, and no cited 'uniqueness theorem' or self-citation is load-bearing. The only in-scope concern is that Sec. 5.3 states 'we conduct all ablation experiments using the test set of PHOENIX14T,' so the reported SOTA margin on that benchmark is at risk of selection bias; this is a correctness/reporting issue, not a derivation-circularity, because the result does not reduce to an input by construction and the CSL-Daily results remain an independent external check. Accordingly, no circular steps are identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard supervised learning and off-the-shelf models; the main free parameters are the contrastive temperature, the loss weight, and LoRA settings, whose values are deferred to the supplementary. The method introduces no new physical or conceptual entities.

free parameters (4)
  • temperature tau in contrastive loss = not reported in main text
    Learnable temperature in L_ALIGN (Eq. 5) scales logits; value fit during pre-training.
  • weight lambda for description-mapper loss = not reported in main text
    Hyperparameter balancing L_ALIGN and L_DM in Eq. 6; value chosen by the authors.
  • LoRA rank for mBART encoder adaptation = not reported in main text
    Low-rank adaptation dimension for the multimodal encoder; authors do not list its value in the main text.
  • frame sampling schedule T = not reported in main text
    Number of frames per video used for visual features and descriptions; affects sequence length.
assumptions (4)
  • domain assumption Pre-trained image-based MLLMs (LLaVA-OneVision 7B) generate sufficiently accurate and relevant descriptions of sign language components from individual frames
    Sec. 3 provides qualitative evidence, but the reliability on test domains is assumed; per-frame descriptions cannot directly capture motion.
  • domain assumption The description mapper (two-layer MLP) can predict BERT description embeddings from ResNet visual features well enough to preserve the benefits at inference
    Eq. 1-2 and the ablated row 6 vs row 4 in Table 3 support this, but it is an architectural assumption.
  • domain assumption Pre-trained BERT, mBART, and ResNet18 weights transfer to the sign language domain
    Model relies on these off-the-shelf frozen/initialized encoders (Sec. 4.2); no domain-specific adaptation is guaranteed.
  • standard math Contrastive InfoNCE and cross-entropy losses are appropriate for aligning and decoding the multimodal features
    Standard training objectives; no novelty is claimed for the losses themselves.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation." pith.science (2026). https://pith.science/paper/RS6YF33Q

@misc{pith2026241116789,
  author       = {Pith},
  title        = {Pith review of: Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RS6YF33Q}},
  note         = {Machine review of arXiv:2411.16789}
}
read the original abstract

Sign language translation (SLT) is a challenging task that involves translating sign language images into spoken language. For SLT models to perform this task successfully, they must bridge the modality gap and identify subtle variations in sign language components to understand their meanings accurately. To address these challenges, we propose a novel gloss-free SLT framework called Multimodal Sign Language Translation (MMSLT), which leverages the representational capabilities of off-the-shelf multimodal large language models (MLLMs). Specifically, we use MLLMs to generate detailed textual descriptions of sign language components. Then, through our proposed multimodal-language pre-training module, we integrate these description features with sign video features to align them within the spoken sentence space. Our approach achieves state-of-the-art performance on benchmark datasets PHOENIX14T and CSL-Daily, highlighting the potential of MLLMs to be utilized effectively in SLT. Code is available at https://github.com/hwjeon98/MMSLT.

Figures

Figures reproduced from arXiv: 2411.16789 by the authors.

Figure 1
Figure 1. Comparison of the proposed MMSLT with GFSLT [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of SL descriptions based on the type of MLLM and prompt. (a) shows responses from Video-LLaVA 7B [ [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. List of prompts. shown in Fig. 2c, LLaVA-OneVision provides descriptions focused solely on SL components without including unre￾lated information. In this experiment, large models, such as LLaVA-OneVision 72B, are excluded from the analysis due to the processing time and resource constraints associated with generating SL descriptions. Prompt Engineering. To effectively leverage MLLMs, it is crucial to select an appr… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: MMSLT overview. MMSLT comprises two modules: [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 38 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am ´elie H ´eliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timoth ´ee Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Mar...

    work page 2024

  3. [3]

    Flamingo: a visual language model for few-shot learning,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Bink...

  4. [4]

    Neural machine translation by jointly learning to align and translate, 2016

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016. 6

    2016

  5. [5]

    Neural sign language trans- lation

    Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, Her- mann Ney, and Richard Bowden. Neural sign language trans- lation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7784–7793, 2018. 1, 2, 6, 7

    work page 2018

  6. [6]

    Sign language transformers: Joint end-to- end sign language recognition and translation

    Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. Sign language transformers: Joint end-to- end sign language recognition and translation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023–10033, 2020. 2, 6, 7

    work page 2020

  7. [7]

    Vlp: A survey on vision-language pre-training

    Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. Machine Intelligence Re- search, 20(1):38–56, 2023. 3

    work page 2023

  8. [8]

    A simple multi-modality transfer learning baseline for sign language translation

    Yutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu, and Stephen Lin. A simple multi-modality transfer learning baseline for sign language translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5120–5130, 2022. 2, 6, 7

    work page 2022

Show all 70 references
  1. [9]

    Two-stream network for sign language recognition and translation

    Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak. Two-stream network for sign language recognition and translation. Advances in Neural Information Processing Systems, 35:17043–17056, 2022. 2, 6, 7

    2022

  2. [10]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision , pages 104–120. Springer,

  3. [11]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, 2024. 3, 8

    2024

  4. [12]

    Factorized learning assisted with large language model for gloss-free sign language translation

    Zhigang Chen, Benjia Zhou, Jun Li, Jun Wan, Zhen Lei, Ning Jiang, Quan Lu, and Guoqing Zhao. Factorized learning assisted with large language model for gloss-free sign language translation. arXiv preprint arXiv:2403.12556,

    arXiv

  5. [13]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3):6,

    2023

  6. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 4

    2009

  7. [15]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. 4, 8

    2019

  8. [16]

    Cross-modal neural sign language translation

    Amanda Cardoso Duarte. Cross-modal neural sign language translation. In Proceedings of the 27th ACM international conference on multimedia, pages 1650–1654, 2019. 1

    2019

  9. [17]

    A token-level contrastive framework for sign language translation

    Biao Fu, Peigen Ye, Liang Zhang, Pei Yu, Cong Hu, Xi- aodong Shi, and Yidong Chen. A token-level contrastive framework for sign language translation. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023. 2, 6, 7

    2023

  10. [18]

    Llms are good sign language translators

    Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani, and Jun Liu. Llms are good sign language translators. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18362–18372, 2024. 1, 2, 6, 7, 8

    2024

  11. [19]

    Multimodal-gpt: A vision and lan- guage model for dialogue with humans

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and lan- guage model for dialogue with humans. arXiv preprint arXiv:2305.04790, 2023. 3 9

    2023 arXiv

  12. [20]

    Deep residual learning for image recognition, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. 4

    2015

  13. [21]

    Egolm: Multi-modal language model of egocentric motions, 2024

    Fangzhou Hong, Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions, 2024. 3

    2024

  14. [22]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations , 2022. 5

    2022

  15. [23]

    Eui Jun Hwang, Sukmin Cho, Huije Lee, Youngwoo Yoon, and Jong C. Park. Universal gloss-level representation for gloss-free sign language translation and production, 2024. 1

    2024

  16. [24]

    An efficient sign language translation using spatial configuration and motion dynamics with llms.arXiv preprint arXiv:2408.10593, 2024

    Eui Jun Hwang, Sukmin Cho, Junmyeong Lee, and Jong C Park. An efficient sign language translation using spatial configuration and motion dynamics with llms.arXiv preprint arXiv:2408.10593, 2024. 1

    2024 arXiv

  17. [25]

    Unsupervised dense information retrieval with con- trastive learning, 2022

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with con- trastive learning, 2022. 8

    2022

  18. [26]

    Scaling up visual and vision-language representa- tion learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representa- tion learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR,

  19. [27]

    Visual alignment pre-training for sign language translation

    Peiqi Jiao, Yuecong Min, and Xilin Chen. Visual alignment pre-training for sign language translation. In Computer Vi- sion – ECCV 2024 , pages 349–367, Cham, 2025. Springer Nature Switzerland. 1, 2, 3, 5, 6, 7

    2024

  20. [28]

    Prior knowledge and memory enriched transformer for sign lan- guage translation

    Tao Jin, Zhou Zhao, Meng Zhang, and Xingshan Zeng. Prior knowledge and memory enriched transformer for sign lan- guage translation. In Findings of the Association for Com- putational Linguistics: ACL 2022 , pages 3766–3775, 2022. 2

    2022

  21. [29]

    Llava-onevision: Easy visual task transfer,

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer,

  22. [30]

    Tspnet: Hier- archical feature learning via temporal semantic pyramid for sign language translation

    Dongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang, Benjamin Swift, Hanna Suominen, and Hongdong Li. Tspnet: Hier- archical feature learning via temporal semantic pyramid for sign language translation. Advances in Neural Information Processing Systems, 33:12034–12045, 2020. 2, 6, 7

    2020

  23. [31]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models, 2024. 3, 8

    2024

  24. [32]

    Align before fuse: Vision and language representation learn- ing with momentum distillation

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation. Advances in neural infor- mation processing systems, 34:9694–9705, 2021. 3

    2021

  25. [33]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. 3

    2023

  26. [34]

    Video-llava: Learning united visual represen- tation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023. 3

    2023 arXiv

  27. [35]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 6

    2004

  28. [36]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3

    2024

  29. [37]

    Univl: A unified video and language pre-training model for multimodal understanding and generation

    Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353, 2020. 3

    2002 arXiv

  30. [38]

    Thang Luong, Hieu Pham, and Christopher D. Manning. Ef- fective approaches to attention-based neural machine trans- lation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, 2015. Association for Compu...

    2015

  31. [39]

    Min and X

    Y . Min and X. Chen. Towards faithful sign language transla- tion, 2023. https://openreview.net/forum?id= mWMJN0vbDF. 7

    2023

  32. [40]

    Mochat: Joints-grouped spatio-temporal grounding llm for multi-turn motion com- prehension and description, 2024

    Jiawei Mo, Yixuan Chen, Rifen Lin, Yongkang Ni, Min Zeng, Xiping Hu, and Min Li. Mochat: Joints-grouped spatio-temporal grounding llm for multi-turn motion com- prehension and description, 2024. 3

    2024

  33. [41]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318,

  34. [42]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023. 3

    2023 arXiv

  35. [43]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  36. [44]

    All you need in sign language production, 2022

    Razieh Rastgoo, Kourosh Kiani, Sergio Escalera, Vassilis Athitsos, and Mohammad Sabokrou. All you need in sign language production, 2022. 1

    2022

  37. [45]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguis- tics, 2019. 8

    2019

  38. [46]

    Sign lan- guage gesture recognition

    Rajat Sharma, Ravi Khapra, and Neeraj Dahiya. Sign lan- guage gesture recognition. International Journal of Recent Research Aspects, 7(2):14–19, 2020. 1

    2020

  39. [47]

    Stokoe, William C

    Jr. Stokoe, William C. Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf. The Journal of Deaf Studies and Deaf Education , 10(1):3– 37, 2005. 1 10

    2005

  40. [48]

    Mul- tilingual translation with extensible multilingual pretraining and finetuning, 2020

    Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. Mul- tilingual translation with extensible multilingual pretraining and finetuning, 2020. 5, 6, 8

    2020

  41. [49]

    Alpaca: A strong, replicable instruction- following model

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Founda- tion Models. https://crfm. stanford. edu/2023/03/13...

    2023

  42. [50]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 3

    2023 arXiv

  43. [51]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 3

    2017

  44. [52]

    The chal- lenges of cross-modal translation: English-to-sign-language translation in the zardoz system

    Tony Veale, Alan Conway, and Br ´ona Collins. The chal- lenges of cross-modal translation: English-to-sign-language translation in the zardoz system. Machine Translation, 13: 81–106, 1998. 1

    1998

  45. [53]

    Stochastic transformer networks with linear competing units: Application to end-to-end sl translation

    Andreas V oskou, Konstantinos P Panousis, Dimitrios Kos- mopoulos, Dimitris N Metaxas, and Sotirios Chatzis. Stochastic transformer networks with linear competing units: Application to end-to-end sl translation. In Proceedings of the IEEE/CVF International Conference on Comput...

    2021

  46. [54]

    Text embeddings by weakly-supervised contrastive pre- training, 2024

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre- training, 2024. 8

    2024

  47. [55]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Jun- yang Lin. Qwen2-vl: Enhancing vision-language model’s ...

    2024

  48. [56]

    Simvlm: Simple visual language model pretraining with weak supervision

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021. 3

    2021 arXiv

  49. [57]

    Sign2gpt: Leveraging large language models for gloss-free sign language translation

    Ryan Wong, Necati Cihan Camgoz, and Richard Bowden. Sign2gpt: Leveraging large language models for gloss-free sign language translation. arXiv preprint arXiv:2405.04164,

    arXiv

  50. [58]

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S. Yu. Multimodal large language models: A sur- vey. In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256, 2023. 2

    2023

  51. [59]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jian- wei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024

  52. [60]

    Filip: Fine-grained interactive language-image pre-training

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 3

    2021 arXiv

  53. [61]

    Simulslt: End-to- end simultaneous sign language translation

    Aoxiong Yin, Zhou Zhao, Jinglin Liu, Weike Jin, Meng Zhang, Xingshan Zeng, and Xiaofei He. Simulslt: End-to- end simultaneous sign language translation. In Proceedings of the 29th ACM International Conference on Multimedia , pages 4118–4127, 2021. 2

    2021

  54. [62]

    Gloss attention for gloss-free sign language translation

    Aoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin, Tao Jin, and Zhou Zhao. Gloss attention for gloss-free sign language translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2551–2562,

  55. [63]

    Sltunet: A simple unified model for sign language translation

    Biao Zhang, Mathias M ¨uller, and Rico Sennrich. Sltunet: A simple unified model for sign language translation. arXiv preprint arXiv:2305.01778, 2023. 1, 2, 6, 7

    2023 arXiv

  56. [64]

    Speechgpt: Empow- ering large language models with intrinsic cross-modal con- versational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empow- ering large language models with intrinsic cross-modal con- versational abilities. arXiv preprint arXiv:2305.11000, 2023. 3

    2023 arXiv

  57. [65]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 3

    2023 arXiv

  58. [66]

    Conditional sentence generation and cross-modal reranking for sign language translation

    Jian Zhao, Weizhen Qi, Wengang Zhou, Nan Duan, Ming Zhou, and Houqiang Li. Conditional sentence generation and cross-modal reranking for sign language translation. IEEE Transactions on Multimedia, 24:2662–2672, 2021. 1, 2, 6

    2021

  59. [67]

    Gloss-free sign language translation: Improving from visual- language pretraining

    Benjia Zhou, Zhigang Chen, Albert Clap ´es, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Gloss-free sign language translation: Improving from visual- language pretraining. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision , pages 2...

    2023

  60. [68]

    Improving sign language translation with monolingual data by sign back-translation

    Hao Zhou, Wengang Zhou, Weizhen Qi, Junfu Pu, and Houqiang Li. Improving sign language translation with monolingual data by sign back-translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 1316–1325, 2021. 2, 6, 7

    2021

  61. [69]

    Spatial-temporal multi-cue network for sign language recog- nition and translation

    Hao Zhou, Wengang Zhou, Yun Zhou, and Houqiang Li. Spatial-temporal multi-cue network for sign language recog- nition and translation. IEEE Transactions on Multimedia , 24:768–779, 2021. 2, 6

    2021

  62. [70]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 3 11

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.