LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation

Bihui Yu; Caijun Jia; Gaowei Wu; Jingxuan Wei; Linzhuang Sun; Qi Chen; Xiangxiang Zhang; Yujun Cai

REVIEW 5 major objections 5 minor 51 references

LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that selectively fine-tuning 40% of a vision-language model's parameters can match or beat full fine-tuning for multilingual multimodal translation.

desk verdict Plausible parameter-efficient recipe, but the 40% claim and the selection protocol are underspecified enough that the efficiency story isn't checkable as written. read the letter →

arxiv 2507.18940 v1 pith:N6ZBPCRC submitted 2025-07-25 cs.CL cs.MM

Jingxuan Wei , Caijun Jia , Qi Chen , Yujun Cai , Linzhuang Sun , Xiangxiang Zhang , Gaowei Wu , Bihui Yu This is my paper · ORCID

classification cs.CLcs.MM

keywords multilingualmultimodalmachinetranslationparameter-efficientfine-tuninglayerselectionneuroncross-lingualinterferencevision-languagemodelsgradientmasking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that adapting a large vision-language model to many language pairs is better done by selecting which parameters to update than by fine-tuning everything. It proposes a two-level selection: first find which transformer layers carry the most task-relevant information, then find which neurons inside those layers are specific to one language pair and which are shared across all pairs. On the M3-Multi30K and M3-AmbigCaps benchmarks, the paper reports that fine-tuning only 40% of the 7-billion-parameter model's parameters slightly exceeds full fine-tuning in average BLEU, and fine-tuning 80% improves the average further. This matters because it points toward adapting one large multimodal system to many languages with a small parameter budget, while also showing which parts of the model absorb the adaptation.

What carries the argument

The load-bearing object is the redundancy-based layer importance score $R_l = (X_l^A \cdot X_l^B)^2 / (\|X_l^A\|^2 \|X_l^B\|^2 + \epsilon)$, where $X_l^A$ and $X_l^B$ are the activations of layer $l$ in the pretrained and fine-tuned models; a low score means the layer changed substantially during fine-tuning and is therefore selected. On top of this, neuron importance $I_n = |A_n \times G_n|$ is aggregated across language pairs, and its variance $\sigma^2(n)$ classifies neurons as language-specific (the pair for which importance is maximal) or language-agnostic (variance below a threshold $\epsilon$). A gradient mask $G'_n = G_n$ if $n \in A \cup S_k$, and $0$ otherwise, restricts updates to the union of agnostic and current-pair-specific neurons. This two-level mask is what lets the model keep most parameters frozen while still adapting per language pair.

What would settle it

Recompute LLaVA-NeuMT with the layer importance scores obtained from a model fine-tuned only on a training split, with the layer fraction and variance cutoff fixed before any test-set evaluation; if the 40%-parameter variant then scores at or below the full-fine-tuning baseline on M3-Multi30K and M3-AmbigCaps, the central claim would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that multilingual interference can be mitigated by selective parameter sharing rather than by sharing everything. The method computes a redundancy-based importance score for each layer by comparing its activations in the pretrained model with its activations in a fine-tuned model: a layer whose activations change a lot is deemed important and kept trainable. Within those layers, each neuron receives an importance score from the product of its activation and gradient, and the variance of that score across language pairs separates language-specific neurons from language-agnostic ones. Only neurons that are either language-agnostic or specific to the current language pair receive gradient updates. The paper reports that with 40% of parameters trainable this selection reaches 53.4 average BLEU on M3-Multi30K versus 53.3 for full fine-tuning, and 58.3 versus 58.0 on M3-AmbigCaps; with 80% trainable it reaches 54.3 and 59.7. It further reports that a 1:9 ratio of specific to agnostic neurons and selection of the top 80% of layers give the best balance.

Load-bearing premise

The whole selection depends on a reference "fine-tuned model" whose activations define which layers are important, and on layer and neuron thresholds that the paper does not fully specify; if that reference is the full fine-tune the method is meant to beat, or if the thresholds were chosen by looking at test results, the reported gains are inflated.

Editorial extensions

If this is right

A 7-billion-parameter multimodal translation model can be adapted to a new language pair by updating only the layers and neurons identified as relevant, cutting the parameter update budget to 40% while keeping or slightly improving BLEU.
The final 20% of layers are not just unnecessary; including them reduces average BLEU, so redundancy removal is part of what improves translation, not merely a cost saving.
Language-agnostic neurons carry more of the multilingual adaptation load than language-specific neurons, since fine-tuning only agnostic neurons outperforms fine-tuning only specific neurons on average.
Layer importance is consistent across the six language pairs studied, so the same layer mask can be computed once and reused for other pairs without per-pair search.
Larger gains appear on the lower-resource pairs (Latvian, Hindi, Turkish), suggesting selective modulation is most valuable where cross-lingual interference is strongest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reference model behind Equation (3) is the fully fine-tuned model, then the 40% parameter budget is a saving for later adaptations, not for the initial fine-tuning: one full fine-tune must be completed first to measure layer importance.
A natural test the paper does not run is whether a layer and neuron mask computed on one set of language pairs transfers to unseen pairs; the reported stability of layer importance across languages suggests it might.
The same two-level selection could be applied to other multilingual multimodal tasks such as image captioning or visual question answering, since the mechanism only assumes that some neurons are shared across languages and some are not.
The parameter budget alone does not measure compute; a complete efficiency picture would include wall-clock training time and memory, because forward and backward passes still run through the full frozen model.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a reasonable incremental idea — pick important layers by activation change after SFT, pick important neurons by activation×gradient variance across languages, then fine-tune only those. The combination for multilingual multimodal MT is new relative to the cited literature, and the experiments cover two datasets and six language pairs, which is a solid amount of work. The paper is clearly written and does not oversell wildly; the limitation section acknowledges fixed thresholds.

What worried me, in the order it bothered me. First, Eq. 3 never says which "fine-tuned model" X_B_l comes from. If that is the fully fine-tuned LLaVA-1.5-SFT baseline, then the pipeline pays for a full SFT pass before selection, and the "40% of parameters" describes only the second stage. That would make the efficiency claim misleading. This is the load-bearing question and it is simply not addressed.

Second, the hyperparameters look selected on test. Figure 3a sweeps the layer fraction and Figure 3b sweeps the neuron ratio; the paper describes no validation split for the M3 datasets. If those curves are computed on the test set, the reported numbers are inflated, and the 1:9 ratio is a fitted parameter, not a discovered regularity. The specific-agnostic distinction itself is also not crisply defined: Eq. 7 and Eq. 8 can overlap, and the paper never states whether the 1:9 ratio means 10% of neurons are updated or 10% are classified as specific within layers that are all updated. The 40% parameter figure is therefore ambiguous: layer count is not parameter count.

Third, the headline BLEU gains are small: 0.1–0.3 on average for the 40% variant, with several language pairs losing to the baseline. No error bars, no repeated runs, no significance test. Given the tuning-on-test concern, I would not trust those numbers as evidence of superiority over full fine-tuning. The 80% variant looks better, but still within the same noise band.

I want to give credit where it is due: the idea of separating language-specific and language-agnostic neurons is well motivated, and the visualizations in Figure 4 are a legitimate attempt to show that the selected components differ across languages. The paper does not fabricate anything; it just leaves too many procedural details unspecified. The citation pattern looks fine — LanderMT is credited as the closest neuron-selection prior work.

Bottom line: this deserves a serious referee, but not acceptance as is. The authors need to say which checkpoint provides X_B_l, run their selection on a validation split, report variance over seeds, and give exact parameter counts. If those are fixed, the method might well stand up. As written, the efficiency and SOTA claims are unverifiable.

Referee Report

5 major / 5 minor

Summary. The paper proposes LLaVA-NeuMT, a method for multilingual multimodal machine translation built on LLaVA-1.5-7B. The method has three components: (i) a layer selection step (Section 3.2) that ranks layers by the squared cosine similarity between activations of the pretrained model and a fine-tuned model, keeping the top alpha fraction; (ii) a neuron classification step (Section 3.3) that labels neurons as language-specific or language-agnostic using importance-score variance across language pairs; and (iii) a gradient masking strategy (Section 3.4) that updates only selected neurons. Experiments on M3-Multi30K and M3-AmbigCaps report BLEU scores for 40% and 80% layer-selection variants, with the abstract claiming that fine-tuning only 40% of parameters surpasses full fine-tuning and achieves state-of-the-art results. The paper also includes ablations on layer fraction, neuron ratio, and visualizations of layer and neuron importance.

Significance. If the central efficiency claim were verified, the paper would make a useful contribution to parameter-efficient multilingual multimodal adaptation: it proposes a concrete layer-and-neuron selection recipe, reports ablations over two hyperparameters, and provides some analysis of which modules matter. The paper also compares against several open- and closed-source multimodal baselines. However, the current evidence does not support the headline claims as stated. The selection protocol in Eq. (3) is underspecified in a way that directly affects whether the method actually saves training cost, the parameter-count accounting is ambiguous, the small BLEU gains over the full fine-tuning baseline lack statistical support, and key hyperparameters appear to be selected using test-set performance. These issues are load-bearing for the paper's central claim, so the manuscript needs substantial revision before the efficiency and SOTA conclusions can be accepted.

major comments (5)

[Section 3.2, Eq. (3)] The definition of R_l requires activations X_B_l from a 'fine-tuned model', but the manuscript never states which checkpoint supplies X_B_l nor on which data split these activations are computed. If X_B_l comes from the fully fine-tuned LLaVA-1.5-SFT model used as the baseline in Tables 1 and 2, then layer selection requires a complete SFT pass before the '40% fine-tuning' stage begins, so the total computational cost and number of parameter updates over the whole pipeline can exceed full fine-tuning. This would directly invalidate the efficiency claim in the abstract. The paper must specify the reference checkpoint, the data split used for selection, and whether selection uses any information from the test sets; otherwise the method is not reproducible and the efficiency claim is not checkable.
[Sections 4.3-4.4 and Figures 3a-3b] The layer fraction alpha and the 1:9 specific-to-agnostic neuron ratio appear to be selected by sweeping on BLEU scores reported in Figure 3a and Figure 3b. If these sweeps are performed on the test sets, then the BLEU numbers in Tables 1 and 2 for the selected configurations are not independent test-set measurements, and the 'SOTA' claim is circular. The paper should report a held-out validation split for hyperparameter selection, or clearly state that the reported test results are for configurations selected without test-set access. The Section 5 limitation statement that 'our current design relies on fixed thresholds' makes this concern more acute, because the thresholds are not derived from a principled criterion but appear to be tuned to the evaluation sets.
[Abstract and Section 3.4, Eq. (9)] The claim 'fine-tuning only 40% of the model parameters' is ambiguous. The 40% label is used for the top 40% of layers, but Eq. (9) also freezes neurons outside A ∪ S_k, so within selected layers some neurons should receive no gradient. If neurons are masked in this way, the true fraction of updated parameters is lower than 40%. If, instead, every neuron in a selected layer belongs to either some S_k or A, then no neuron is actually frozen and the 40% figure simply measures the layer fraction. The paper should report the exact number of trainable parameters for the 40% and 80% variants, including whether the vision encoder, connector, embeddings, and LM head are updated, and should reconcile this accounting with the claim that the method is more efficient than full fine-tuning.
[Tables 1-2] The empirical support for 'surpasses full fine-tuning' is thin. The 40% variant beats the LLaVA-1.5-SFT baseline by 0.1 BLEU on average on M3-Multi30K (53.4 vs. 53.3) and by 0.3 on M3-AmbigCaps (58.3 vs. 58.0), while losing on several individual language pairs: on M3-Multi30K it is below baseline for Cs and Hi, and on M3-AmbigCaps it is below baseline for Cs, Lv, and Hi. No standard deviations, confidence intervals, significance tests, or multiple-seed results are reported. A single run with these margins cannot substantiate a claim of surpassing full fine-tuning. The authors should provide variance estimates, paired significance tests, or multiple seeds, and should temper the abstract if the differences are not significant.
[Section 3.3, Eqs. (5)-(8)] The neuron classification definitions are underspecified in ways that affect the method. Eq. (5) aggregates importance scores over T training samples and K language pairs but does not state whether I_k^n is the mean over samples, the sum, or something else. Eq. (7) defines S_k as the set where I_k^n is the maximum over languages, but this condition can assign the same neuron to multiple language pairs when scores are tied, and it leaves unclassified any neuron that is never the maximum. Eq. (8) uses a variance threshold epsilon whose value is never given. The authors should define the aggregation precisely, state how ties are broken, specify epsilon or describe how it is set, and report the resulting sizes of S_k and A; otherwise the gradient masking in Eq. (9) cannot be implemented from the paper.

minor comments (5)

[Throughout] The model name is spelled 'LLaV A' in the abstract and main text in several places; it should be 'LLaVA' consistently.
[Section 4.5 and Section 4.6] The layer counts are inconsistent with the LLaVA-1.5-7B architecture. Section 4.5 refers to 'the first 250 layers (approximately 80% of the model depth)', and Section 4.6 refers to 'the top 40% of layers (108 layers)', but LLaVA-1.5-7B has 32 transformer layers. Please clarify what unit is being counted (e.g., parameter tensors, MLP blocks, or submodules) and make the layer counts consistent with the architecture.
[Section 4.3 vs. Abstract] The paper says in Section 4.3 that selecting 80% of layers gives the best balance, and Tables 1 and 2 show the 80% variant outperforms the 40% variant on both datasets, while the abstract emphasizes the 40% variant. This tension should be addressed explicitly, including whether the 40% variant or the 80% variant is the recommended configuration and what 'fine-tuning only 40%' means in the abstract.
[Figure 3b] The x-axis labels '0:1', '1:9', etc., are unclear without reading the caption; please clarify in the axis label that these are specific-to-agnostic neuron ratios.
[Tables 1-2] The tables do not report model parameter counts or training cost for the proposed variants, which makes it hard to evaluate the efficiency claim. Please include the number of trainable parameters and, if possible, training time or FLOPs.

Circularity Check

2 steps flagged · score 6.0 of 10

Layer mask is read from the fully fine-tuned baseline it claims to avoid, and the neuron ratio is chosen by test-BLEU sweep and then reported as independent support.

fitted input called prediction [Section 3.2, Eq. (3)]
"The importance of each layer is assessed based on activation similarity before and after supervised fine-tuning (SFT)... Given activations from layer l in the pretrained model (X A l ) and the fine-tuned model (X B l ), the redundancy-based importance score R l is computed as: R l = (X A l · X B l )^2 / (||X A l||^2 ||X B l||^2 + ε)... A lower R l value suggests that a layer undergoes significant adaptation during fine-tuning, indicating its importance for the translation task."

The paper never specifies which checkpoint provides X_B_l. The only fully fine-tuned model in the paper is the LLaVA-1.5-SFT(default) baseline (Tables 1-2) that LLaVA-NeuMT (40%) is claimed to surpass. On that natural reading, the layer mask is read off the very full-fine-tuning result the method claims to avoid; the 'fine-tuning only 40% of the model parameters' claim omits this prerequisite full fine-tuning pass. The selection is therefore fitted to the output of the full-fine-tuning process rather than derived from the pretrained model alone, and the efficiency claim is not checkable as stated.
fitted input called prediction [Section 4.4 and Section 4.6, Figure 3b]
"As shown in Figure 3b, increasing the proportion of specific neurons initially improves BLEU scores, peaking at a 1:9 ratio... From a distribution perspective, the first 10% of neurons in each layer (dashed lines) display high variance, while the remaining 90% (solid lines) maintain stable scores. This supports our choice of a 1:9 ratio between specific and agnostic neurons, ensuring an optimal balance between language adaptability and cross-lingual generalization."

The 1:9 specific-to-agnostic ratio is first selected by sweeping BLEU over ratio values in Figure 3b, and the layer fraction α is likewise selected from the Figure 3a test-BLEU sweep. Section 4.6 then cites variance statistics of the already-selected configuration as evidence that 1:9 is the right balance. The claimed 'insight' that the first 10% of neurons are specific and 90% are agnostic is a restatement of the chosen hyperparameter, not an independent confirmation from the mechanism; the evidentiary order is inverted.

full rationale

The central efficiency claim is undercut by one load-bearing ambiguity: Eq. (3) defines layer importance using activations X_B_l of a 'fine-tuned model' without stating how that model was produced. If X_B_l is the fully fine-tuned LLaVA-1.5-SFT baseline that the method is supposed to beat, then the 40% mask is a function of the full-fine-tuning result, so the pipeline implicitly requires full fine-tuning before any parameter-efficient run can begin. The paper also tunes the layer fraction and the 1:9 neuron ratio on test BLEU sweeps (Figure 3a-b) and then presents the chosen configuration as an empirical discovery about layer/neuron importance (Sections 4.3-4.6), which is post-hoc fitting rather than independent derivation. These issues make the headline 'surpasses full fine-tuning while fine-tuning only 40% of parameters' partially circular as stated. The remainder of the paper is a standard empirical comparison with external baselines and does not depend on self-citation, so the circularity is localized to the selection mechanism and the efficiency claim.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central method depends on two empirically motivated importance heuristics, three tuned hyperparameters (alpha, epsilon, ratio), and an unspecified reference model for layer scoring. These are the main burdens on the reader.

free parameters (3)

Layer selection fraction alpha = 0.4 (reported) and 0.8 (best per Figure 3a)
Top fraction of layers retained for fine-tuning; varied in Figure 3a and chosen based on translation BLEU.
Neuron variance threshold epsilon = not reported
Threshold in Eq. 8 for declaring a neuron language-agnostic; no value or selection method given.
Specific-to-agnostic neuron ratio = 1:9
Ratio used in final model; determined by sweeping ratios in Figure 3b and Section 4.4.

assumptions (5)

domain assumption Pretrained LLaVA-1.5-7B provides representations suitable for multilingual multimodal translation.
The method fine-tunes this backbone and assumes its English-centric vision-language knowledge transfers to the six target languages.
domain assumption Activation similarity between pretrained and fine-tuned models (Eq. 3) ranks layer importance for the translation task.
Heuristic borrowed from pruning literature (citations [21,44]); no proof that cosine similarity of activations identifies layers critical for translation.
domain assumption Neuron importance defined as |activation times gradient| (Eq. 4) identifies language-relevant neurons.
Standard saliency heuristic, cited to [50]; assumed without justification for this setting.
ad hoc to paper Language-specific and language-agnostic neurons are separable by variance of importance scores across language pairs (Eqs. 7-8).
The use of variance thresholding to split neuron types is introduced in this paper and not derived from any theory.
ad hoc to paper A fully fine-tuned reference model is available for computing layer importance in Eq. 3.
The paper does not state which model serves as X_B_l or how it was trained; without this, the method cannot be run.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation." pith.science (2026). https://pith.science/paper/N6ZBPCRC

@misc{pith2026250718940,
  author       = {Pith},
  title        = {Pith review of: LLaVA-NeuMT: Selective Layer-Neuron Modulation for Efficient Multilingual Multimodal Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N6ZBPCRC}},
  note         = {Machine review of arXiv:2507.18940}
}

read the original abstract

Multimodal Machine Translation (MMT) enhances translation quality by incorporating visual context, helping to resolve textual ambiguities. While existing MMT methods perform well in bilingual settings, extending them to multilingual translation remains challenging due to cross-lingual interference and ineffective parameter-sharing strategies. To address this, we propose LLaVA-NeuMT, a novel multimodal multilingual translation framework that explicitly models language-specific and language-agnostic representations to mitigate multilingual interference. Our approach consists of a layer selection mechanism that identifies the most informative layers for different language pairs and a neuron-level adaptation strategy that dynamically selects language-specific and agnostic neurons to improve translation quality while reducing redundancy. We conduct extensive experiments on the M3-Multi30K and M3-AmbigCaps datasets, demonstrating that LLaVA-NeuMT, while fine-tuning only 40\% of the model parameters, surpasses full fine-tuning approaches and ultimately achieves SOTA results on both datasets. Our analysis further provides insights into the importance of selected layers and neurons in multimodal multilingual adaptation, offering an efficient and scalable solution to cross-lingual adaptation in multimodal translation.

Figures

Figures reproduced from arXiv: 2507.18940 by the authors.

**Figure 1.** Challenges in MMT. Machine Translation (MNMT) has made progress in text-only translation by leveraging cross-lingual parameter sharing, evolving from simple parameter sharing to more sophisticated approaches like adaptive scheduling and language-specific modules[16, 33, 10]. Recently, Mixture-of-Experts (MoE) models have attempted to dynamically allocate computational resources across languages, but often struggle w… view at source ↗

**Figure 2.** LLaVA-NeuMT Model Architecture. 3 Methodology 3.1 Multimodal Machine Translation Multimodal machine translation extends traditional machine translation by incorporating visual information to enhance contextual understanding. Given a source sentence Xs in language s, a corresponding image I, and a target language t, the objective is to generate a translated sentence Y t that preserves the semantics of the source sent… view at source ↗

**Figure 3.** Translation performance under different model configurations. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Layer and neuron analysis on M3-Multi30K. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 41 canonical work pages

[1]

Achiam, S

J. Achiam, S. Adler, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

Aharoni, M

R. Aharoni, M. Johnson, and O. Firat. Massively multilingual neural machine translation. In NAACL, pages 3874–3884, 2019

work page 2019
[3]

G. Chen, L. Hou, Y . Chen, W. Dai, L. Shang, X. Jiang, Q. Liu, J. Pan, and W. Wang. mclip: Multilingual clip via cross-lingual transfer. In ACL, pages 13028–13043, 2023

work page 2023
[4]

L. Chen, S. Ma, D. Zhang, et al. On the pareto front of multilingual neural machine translation. NeurIPS, 36, 2024

work page 2024
[5]

Z. Chen, J. Wu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024

work page 2024
[6]

Z. Chen, F. Yin, X.-Y . Zhang, Q. Yang, and C.-L. Liu. Cross-lingual text image recognition via multi-task sequence to sequence learning. In ICPR, pages 3122–3129, 2021

work page 2021
[7]

Dabre, C

R. Dabre, C. Chu, and A. Kunchukuttan. A survey of multilingual neural machine translation. ACM Computing Surveys, 53(5):1–38, 2020

work page 2020
[8]

A. Fan, S. Bhosale, H. Schwenk, et al. Beyond english-centric multilingual machine translation. JMLR, 22(107):1–48, 2021

work page 2021

Show all 51 references

[9]

Fedus, B

W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 23(120):1–39, 2022

2022
[10]

X. Feng, X. Geng, B. Li, B. Qin, et al. Towards higher pareto frontier in multilingual machine translation. In ACL, 2023

2023
[11]

H. Guo, J. Liu, H. Huang, et al. Lvp-m3: Language-aware visual prompt for multilingual multimodal machine translation. In EMNLP, pages 2862–2872, 2022

2022
[12]

Hinami, S

R. Hinami, S. Ishiwatari, et al. Towards fully automated manga translation. In AAAI, volume 35, pages 12998–13008, 2021

2021
[13]

Huang, F

K. Huang, F. Mo, H. Li, et al. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936, 2024

2024 arXiv
[14]

P. Jain, O. Firat, Q. Ge, and S. Liang. Image translation network. 2021

2021
[15]

Javed, H

A. Javed, H. Zan, O. Mamyrbayev, M. Abdullah, et al. Transformer-based re-ranking model for enhancing contextual and syntactic translation in low-resource neural machine translation. Electronics, 14(2):243, 2025

2025
[16]

S. Jean, O. Firat, and M. Johnson. Adaptive scheduling for multi-task learning. arXiv preprint arXiv:1909.06434, 2019

1909 arXiv
[17]

Klouchek and R

B. Klouchek and R. T. Batista-Navarro. Bulgarian grammar error correction with data augmen- tation and machine translation techniques. In ICNLSP 2024, pages 365–376, 2024

2024
[18]

Z. Lan, J. Yu, X. Li, W. Zhang, J. Luan, B. Wang, D. Huang, and J. Su. Exploring better text image translation with multimodal codebook. In ACL, pages 3479–3491, 2023

2023
[19]

J. Li, D. Ataman, and R. Sennrich. Vision matters when it should: Sanity checking multimodal machine translation models. In EMNLP, pages 8556–8562, 2021. 10

2021
[20]

S. Li, X. Wei, S. Zhu, et al. Mmnmt: Modularizing multilingual neural machine translation with flexibly assembled moe and dense blocks. In EMNLP, pages 4978–4990, 2023

2023
[21]

W. Li, L. Li, M. G. Lee, and S. Sun. Adaptive layer sparsity for large language models via activation correlation assessment. In NeurIPS, 2024

2024
[22]

Liang, F

Y . Liang, F. Meng, J. Wang, et al. Continual learning with semi-supervised contrastive distilla- tion for incremental neural machine translation. In ACL, pages 10914–10928, 2024

2024
[23]

H. Liu, C. Li, Y . Li, and Y . J. Lee. Improved baselines with visual instruction tuning. InCVPR, pages 26296–26306, 2024

2024
[24]

J. Liu, H. Qin, Y . Wu, J. Guo, et al. Coupleface: Relation matters for face recognition distillation. In ECCV, pages 683–700, 2022

2022
[25]

C. Ma, X. Han, L. Wu, Y . Zhang, Y . Zhao, Y . Zhou, and C. Zong. Modal contrastive learning based end-to-end text image machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

2024
[26]

C. Ma, Y . Zhang, M. Tu, X. Han, L. Wu, Y . Zhao, and Y . Zhou. Improving end-to-end text image translation from the auxiliary text translation task. In ICPR, pages 1664–1670, 2022

2022
[27]

C. Ma, Y . Zhang, M. Tu, Y . Zhao, Y . Zhou, and C. Zong. Ccim: Cross-modal cross-lingual interactive image translation. In EMNLP 2023, pages 4959–4965, 2023

2023
[28]

C. Ma, Y . Zhang, M. Tu, Y . Zhao, Y . Zhou, and C. Zong. E2timt: Efficient and effective modal adapter for text image machine translation. In ICDAR, pages 70–88, 2023

2023
[29]

C. Ma, Y . Zhang, M. Tu, Y . Zhao, Y . Zhou, and C. Zong. Multi-teacher knowledge distillation for end-to-end text image machine translation. In ICDAR, pages 484–501, 2023

2023
[30]

Mansimov, M

E. Mansimov, M. Stern, M. X. Chen, O. Firat, J. Uszkoreit, and P. Jain. Towards end-to-end in-image neural machine translation. In NLPBT, pages 70–74, 2020

2020
[31]

Nguyen, M

D.-K. Nguyen, M. Assran, U. Jain, M. R. Oswald, C. G. Snoek, and X. Chen. An image is worth more than 16x16 patches: Exploring transformers on individual pixels. arXiv preprint arXiv:2406.09415, 2024

2024 arXiv
[32]

Nimma, V

D. Nimma, V . S. Srinivas, S. S. Gupta, H. Nair, R. L. Devi, and B. K. Bala. Comparative analysis of deep learning models for multilingual language translation. In SLAAI-ICAI, pages 1–6, 2024

2024
[33]

X. Pan, M. Wang, L. Wu, and L. Li. Contrastive learning for many-to-many multilingual neural machine translation. In ACL, pages 244–258, 2021

2021
[34]

R. Peng, Y . Zeng, and J. Zhao. Distill the image to nowhere: Inversion knowledge distillation for multimodal machine translation. In EMNLP, pages 2379–2390, 2022

2022
[35]

Philip, A

J. Philip, A. Berard, M. Gallé, and L. Besacier. Monolingual adapters for zero-shot neural machine translation. In EMNLP, pages 4465–4470, 2020

2020
[36]

Poppi, Z.-X

S. Poppi, Z.-X. Yong, Y . He, B. Chern, H. Zhao, A. Yang, and J. Chi. Towards understanding the fragility of multilingual llms against fine-tuning attacks. arXiv preprint arXiv:2410.18210, 2024

2024 arXiv
[37]

Shaham, M

U. Shaham, M. Elbayad, V . Goswami, O. Levy, and S. Bhosale. Causes and cures for interference in multilingual translation. In ACL, pages 15849–15863, 2023

2023
[38]

T. Su, S. Liu, and S. Zhou. Rtnet: An end-to-end method for handwritten text image translation. In ICDAR, pages 99–113, 2021

2021
[39]

S. Tan, D. Wu, and C. Monz. Neuron specialization: Leveraging intrinsic task modularity for multilingual machine translation. arXiv preprint arXiv:2404.11201, 2024. 11

2024 arXiv
[40]

Tayir, L

T. Tayir, L. Li, B. Li, J. Liu, and K. A. Lee. Encoder-decoder calibration for multimodal machine translation. IEEE Transactions on Artificial Intelligence, 2024

2024
[41]

G. Team, P. Georgiev, , et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

2024 arXiv
[42]

Y . Tian, X. Li, Z. Liu, Y . Guo, and B. Wang. In-image neural machine translation with segmented pixel sequence-to-sequence model. In EMNLP, pages 15046–15057, 2023

2023
[43]

P. Wang, S. Bai, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

2024 arXiv
[44]

Q. Wang, X. Yang, F. Feng, X. Geng, et al. Cluster-learngene: Inheriting adaptive clusters for vision transformers. In NeurIPS, 2024

2024
[45]

X. Wang, Y . Tsvetkov, and G. Neubig. Balancing training for multilingual neural machine translation. In ACL, pages 8526–8537, 2020

2020
[46]

Y . Wang, L. Wang, Q. Zhou, Z. Wang, H. Li, G. Hua, and W. Tang. Multimodal llm enhanced cross-lingual cross-modal retrieval. In ACM MM, pages 8296–8305, 2024

2024
[47]

J. Wei, L. Sun, Y . Leng, X. Tan, B. Yu, and R. Guo. Sentence-level or token-level? a comprehensive study on knowledge distillation. In K. Larson, editor, IJCAI, pages 6531–6540, 8 2024

2024
[48]

Y . Yao, T. Yu, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

2024 arXiv
[49]

Zhang, A

B. Zhang, A. Bapna, R. Sennrich, and O. Firat. Share or not? learning to schedule language- specific capacity for multilingual translation. In ICLR, 2021

2021
[50]

S. Zhu, L. Pan, B. Li, and D. Xiong. Landermt: Dectecting and routing language-aware neurons for selectively finetuning llms to machine translation. In ACL, pages 12135–12148, 2024

2024
[51]

S. Zhu, S. Xu, H. Sun, L. Pan, M. Cui, J. Du, R. Jin, A. Branco, D. Xiong, et al. Multilingual large language models: A systematic survey. arXiv preprint arXiv:2411.11072, 2024. 12 A Broader Impact This work addresses the important challenge of multilingual multimodal machine ...

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Achiam, S

J. Achiam, S. Adler, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[2] [2]

Aharoni, M

R. Aharoni, M. Johnson, and O. Firat. Massively multilingual neural machine translation. In NAACL, pages 3874–3884, 2019

work page 2019

[3] [3]

G. Chen, L. Hou, Y . Chen, W. Dai, L. Shang, X. Jiang, Q. Liu, J. Pan, and W. Wang. mclip: Multilingual clip via cross-lingual transfer. In ACL, pages 13028–13043, 2023

work page 2023

[4] [4]

L. Chen, S. Ma, D. Zhang, et al. On the pareto front of multilingual neural machine translation. NeurIPS, 36, 2024

work page 2024

[5] [5]

Z. Chen, J. Wu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024

work page 2024

[6] [6]

Z. Chen, F. Yin, X.-Y . Zhang, Q. Yang, and C.-L. Liu. Cross-lingual text image recognition via multi-task sequence to sequence learning. In ICPR, pages 3122–3129, 2021

work page 2021

[7] [7]

Dabre, C

R. Dabre, C. Chu, and A. Kunchukuttan. A survey of multilingual neural machine translation. ACM Computing Surveys, 53(5):1–38, 2020

work page 2020

[8] [8]

A. Fan, S. Bhosale, H. Schwenk, et al. Beyond english-centric multilingual machine translation. JMLR, 22(107):1–48, 2021

work page 2021