{"id":"8a2c55f9-1c76-4821-a196-0ecf18327ae1","arxiv_id":"2506.11820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"The authors introduce AibTrans, a multilingual image-text translation benchmark, show that common translation metrics mislead on dense images, and find that balanced multilingual fine-tuning preserves generalization better than single-pair fine-tuning.","lead":"This paper builds a human-checked dataset and benchmark for translating text that appears inside images, testing many commercial and open-source vision-language models. It also proposes a new scoring method that adjusts translation metrics by how dense the text is in an image.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adaptation claim is built on a confounded comparison: single-pair OCRMT30K fine-tuning differs from balanced MIT-10M fine-tuning in data volume, domain, number of directions, and reference style, so the observed degradation and gains cannot yet be attributed to language-pair balance.","rationale":"The stress-test pass should focus on what would falsify the abstract's most consequential claim: that a balanced multilingual fine-tuning strategy adapts LVLMs to VLT without sacrificing generalization. The design in Section 5.2 (Tables 4 and 5) does not control for training set size, number of directions, data domain, or reference style. The OCRMT30K SFT uses roughly 30K examples of one direction; the balanced SFT uses hundreds or thousands per direction from MIT-10M. If the balanced SFT helps mainly because more diverse data at a comparable or larger volume helps any fine-tuning, or because the model learns to imitate GPT-4's MIT-10M reference style, the 'balance' story is not established. The paper's own critique of MIT-10M references in Section 2.2 makes the reference-style confound particularly sharp. Table 6 may offer held-out evidence on AibTrans, but its LoRA rows do not specify the sampling condition, so it cannot currently resolve the issue. The DA Score concern raised by the reader is real — weights are calibrated on GPT-4o outputs and the density intervals overlap — but it is secondary to the adaptation claim: Tables 4 and 5 already support the adaptation finding with standard metric averages, so DA Score invalidity would weaken the evaluation contribution without directly overturning the adaptation claim. The proposed controlled experiment would settle the causal attribution. No independent artifacts or code are released, so this is not currently checkable by third parties. Given the control gap, the reader's CONDITIONAL verdict is appropriate; this stress-test does not change it.","tokens_in":20525,"tokens_out":12545,"duration_ms":108298,"concrete_test":"Run a volume- and domain-matched LoRA experiment on MIT-10M with three conditions: (A) 1,000 examples from one high-resource direction (e.g., zh-to-en), (B) the Table 5 balanced condition (1,000 examples per direction), and (C) 1,000 examples from a low-resource direction (e.g., hi-to-en). Evaluate all conditions on OCRMT30K, MTIT6, and AibTrans using BLEU, CHRF++, BERTScore, COMET, and human ratings on a 100-sample subset of AibTrans. If condition A does not degrade relative to baseline, or if B does not outperform A when volume is matched, the paper's attribution of degradation and recovery to language-pair balance is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in the abstract and Section 5.2 — that fine-tuning on a high-resource pair degrades cross-lingual performance and that balanced multilingual fine-tuning repairs it — rests on a two-cell comparison with multiple simultaneously varying factors. Table 4 fine-tunes LoRA on all ~30K examples of OCRMT30K, a single Chinese-to-English direction (Section 2.1), while Table 5 fine-tunes on 500 or 1000 examples per direction sampled from MIT-10M (potentially ~52K-104K examples across 104 directions). The training set size, number of directions, input domain, and reference source all change at once; nothing isolates 'language-pair balance.' Additionally, Section 2.2 explicitly warns that MIT-10M references are GPT-4 generated and stylistically separable from human translations, yet Table 5 measures the largest gains on MIT-10M, so the improvement may reflect mimicry of those machine references rather than better VLT ability. The paper also mislabels OCRMT30K as 'English-to-Chinese' in Section 5.2 (contradicting Section 2.1), and Table 4 reports only dataset-level aggregates, so the assertion of degradation 'across both high- and low-resource translation directions' is not actually shown. Table 6 may provide held-out evidence on AibTrans, but its LoRA rows do not specify which fine-tuning condition they correspond to, so it cannot currently resolve the confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a three-part study of multilingual vision-language translation (VLT): it critiques existing datasets (MIT-10M, OCRMT30K, MTIT6) through LLM-based quality ratings and OCR error analysis; it introduces AibTrans, a human-verified parallel dataset from Chinese into seven target languages; and it benchmarks 11 commercial and 6 open-source models across end-to-end and cascaded architectures. On the evaluation side, the paper proposes Density-Aware Scoring (DA Score), a weighted combination of BLEU, CHRF++, BERTScore, and COMET with weights meant to reflect metric-human correlation within density categories defined by bounding-box count and token length. Finally, the paper claims that fine-tuning on a single high-resource pair (OCRMT30K) degrades cross-lingual performance, while a balanced multilingual fine-tuning strategy sampling 500-1000 examples per direction from MIT-10M adapts LVLMs to VLT without sacrificing generalization.","tokens_in":20922,"tokens_out":9714,"duration_ms":82778,"significance":"If the claims hold, the paper makes several useful contributions: a concrete quantitative account of OCR error rates in existing VLT data (Fig. 3), a parallel and culturally grounded evaluation resource in AibTrans, a broad model comparison that includes a careful OCR-dependency analysis and the counterintuitive finding that strict instructions can hurt reasoning models (Table 2), and a cost-performance analysis in Appendix B. The empirical breadth is a real strength: 17 models across three datasets with multiple metrics, including machine-checkable tables. However, the two headline methodological contributions are currently under-evidenced. The DA Score is under-specified (weights not reported, no held-out validation) and the balanced fine-tuning conclusion rests on a confounded two-cell comparison, so the paper's central claims go beyond what the present experiments establish.","major_comments":[{"comment":"The DA Score weights alpha, beta, lambda, phi are never reported; the text says only that they are derived from metric-human correlations within each density category. No per-category weight values, no normalization rule, and no held-out validation are provided. Figure 8 then shows that the DA Score distribution 'aligns more closely with the human score curve' on the same GPT-4o translations and the same datasets (MIT-10M, OCRMT30K, MTIT6) used to fit the weights; this is in-sample fitting, and the transfer of the weights to other models' outputs (as used in Tables 6 and Appendix B) is assumed but never tested. The number of human ratings, annotator count, inter-annotator agreement, and per-cell sample sizes behind Table 3 are also omitted. In addition, the text in Section 4.2 claims that BLEU, CHRF++, BERTScore, and COMET 'demonstrate the highest correlations (all > 0.4)', which is contradicted by Table 3 itself (CHRF++ low: 0.2965; BERTScore medium: 0.2650). These details are load-bearing because DA Score is the measure used to support the fine-tuning conclusions in Sections 5.2-5.3.","section":"Section 4.2"},{"comment":"The three density categories are given with overlapping intervals: Low Density is <=10 bounding boxes and 1-30 tokens; Medium Density is 5-20 bounding boxes and 30-50 tokens; High Density is 8-30 bounding boxes and 50-90 tokens. A sample with 6 boxes and 30 tokens satisfies both Low and Medium, and one with 8 boxes and 50 tokens satisfies both Medium and High. Because the DA weights in Eq. (1) are selected per density category, the assignment rule determines the score, and as written the rule is ambiguous. The K-means description ('clustering over these two dimensions') does not state how the two dimensions were scaled, how the number of clusters was chosen, or how the reported intervals were derived, so the taxonomy is not reproducible as specified.","section":"Section 4.2, density taxonomy"},{"comment":"The claim that fine-tuning on a high-resource pair degrades cross-lingual performance while balanced multilingual fine-tuning repairs it rests on a two-cell comparison in which training set size, number of directions, domain, and reference source change simultaneously. Table 4 fine-tunes on roughly 30K OCRMT30K examples in a single direction, whereas Table 5 fine-tunes on 500 or 1000 examples per direction sampled from MIT-10M (up to ~104 directions); nothing isolates 'language-pair balance.' Additionally, Section 2.2 and Figure 4 show that MIT-10M references are GPT-4-generated and stylistically separable from human translations, yet Table 5 reports the largest gains on MIT-10M itself, so the improvements may partly reflect imitation of the machine reference style rather than better VLT ability. Table 4 also reports only dataset-level aggregates, so the assertion of degradation 'across both high- and low-resource translation directions' is not actually exhibited; in fact the in-domain OCRMT30K scores improve after fine-tuning (e.g., Qwen2.5-VL-7B BLEU 17.78 to 25.08).","section":"Section 5.2, Tables 4-5"},{"comment":"Section 5.2 describes OCRMT30K as an 'English-to-Chinese dataset,' which contradicts Section 2.1 and Table 8, where OCRMT30K is defined as containing only Chinese-to-English translations. Since the Table 4 experiment is the entire empirical basis for the high-resource-pair degradation claim, the direction of that pair must be stated correctly and the interpretation of the in-domain gains updated accordingly.","section":"Section 5.2, OCRMT30K direction"},{"comment":"The 'SFT with LoRA' rows in Table 6 do not state which fine-tuning condition they correspond to (OCRMT30K as in Table 4, or balanced MIT-10M sampling with 500 or 1000 examples per direction as in Table 5). The surrounding text credits the gains to the balanced multilingual strategy with 1000 samples, but the table as printed cannot confirm that, and the LoRA row values differ substantially from Tables 4 and 5 because Table 6 is evaluated on AibTrans while Tables 4-5 report on MIT-10M, OCRMT30K, and MTIT6. The fine-tuning condition and evaluation set must be stated explicitly in the table or caption for this held-out evidence to be usable.","section":"Section 5.3, Table 6"}],"minor_comments":[{"comment":"The header 'Collabrative' is a typo for 'Collaborative', and the same word is misspelled in the table's first column; the text in Section 3.4 uses the correct spelling.","section":"Table 2"},{"comment":"The dataset scale is reported inconsistently: Section 2.3 says 630 images were selected for annotation, Section 5.1 says the benchmark contains 100 images and 6993 sentence-level instances, and Table 8 lists 700/7000. The relationship between these numbers (e.g., whether the benchmark uses a 100-image subset) should be clarified.","section":"Section 2.3 vs Section 5.1 and Table 8"},{"comment":"The claim that the DA Score distribution aligns more closely with the human score distribution is supported only by a visual comparison; a quantitative agreement measure (e.g., correlation, RMSE, or KL divergence between the distributions) should be reported.","section":"Figure 8"},{"comment":"The phrase 'from 8 to 13 translation directions in MIT-10M' is ambiguous; since MIT-10M has 8 source and 13 target languages, the intended meaning is 104 directed language pairs, and this should be stated explicitly.","section":"Section 5.2, sampling description"},{"comment":"The notation for BERTScore is inconsistent: it is called BS-F1 in the tables, BERTScore in the text, and BERT-F1 in Eq. (1); please unify the notation.","section":"Eq. (1) and Tables 1-6"},{"comment":"The model counts in the abstract (11 commercial, 6 open-source) do not map transparently to Table 1, which includes w/ OCR and w/o OCR variants as well as language-only cascaded models; the counting convention should be stated.","section":"Abstract and Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable with major revisions, but the abstract currently overstates both methodological contributions relative to the evidence. The DA Score circularity and the fine-tuning confound are the two load-bearing issues; if the authors cannot supply held-out validation for DA Score or control experiments that vary data volume and balance independently, the 'notably' claim in the abstract should be softened or removed. I also note internal inconsistencies (OCRMT30K direction, AibTrans image/instance counts) that suggest the manuscript was assembled hastily; these should be checked carefully in revision. The dataset and benchmarking contributions are within scope for the venue and have independent value even if the two headline claims are scaled back."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the dataset is a real contribution, but the two headline claims—DA Score and balanced fine-tuning—are not yet supported by the experiments as designed.\n\nWhat's new and useful: AibTrans is a genuine attempt at a clean evaluation set: manually corrected OCR and parallel human translations from Chinese into seven languages, with attention to cultural appropriateness. The benchmark across 17 models on three existing datasets is thorough, and the findings on prompt style are concrete: strict instructions degrade reasoning models, while anti-hallucination prompts help end-to-end LVLMs. Those results will be useful to practitioners regardless of the metric problems.\n\nWhere it gets shaky: the DA Score is a weighted average of BLEU, CHRF++, BERTScore, and COMET, with weights said to be derived from metric-human correlations, but the weights are never reported. The validation is in-sample: the same human ratings of GPT-4o outputs used to set the weights are then used to show the DA Score aligns better with human scores. No held-out set, no description of how weights are computed from the correlations. Density categories also overlap (Low: ≤10 boxes / 1-30 tokens; Medium: 5-20 boxes / 30-50 tokens; High: 8-30 boxes / 50-90 tokens), so the taxonomy isn't clean.\n\nThe bigger problem is the fine-tuning claim. The paper says fine-tuning on a high-resource pair degrades cross-lingual performance and balanced multilingual fine-tuning repairs it, but the evidence is a two-cell comparison: LoRA on all ~30K examples of OCRMT30K (single Chinese→English direction) vs. 500-1000 examples per direction from MIT-10M (up to 104K examples across 104 directions). Training volume, number of directions, domain, and reference style all change at once. Nothing isolates language-pair balance. Worse, Section 5.2 calls OCRMT30K \"English-to-Chinese,\" contradicting Section 2.1 where it is Chinese-to-English, and Table 6's LoRA rows don't say which condition they correspond to, so they can't resolve the confound. The gains are also largest on MIT-10M, whose references are GPT-4 generated—so the balanced model may simply be mimicking those machine references.\n\nOther inconsistencies: AibTrans is described as 630 images in Section 2.3, 100 images and 6,993 instances in Section 5.1, and \"700/7000\" in Table 8. No code or data is released.\n\nBottom line: this deserves a serious referee—the dataset and benchmark are worth curating, and the flaws are fixable. But the current version should be a major revision: report the DA Score weights, validate on a held-out set, and redo the fine-tuning experiment with matched data volume and domain. I'd also require the dataset and code to be released.","headline":"The dataset and prompt-style findings are worth your time; the DA Score and balanced fine-tuning claim need another pass.","tokens_in":21414,"tokens_out":3401,"would_cite":false,"duration_ms":29891,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning vision-language models on a single high-resource language pair hurts their other languages; a balanced multilingual sample of about 1,000 examples per direction fixes the loss.","keywords":["vision-language translation","large vision-language models","multilingual evaluation","AibTrans dataset","OCR correction","density-aware evaluation","multilingual fine-tuning","cross-lingual generalization"],"falsifier":"Recompute the metric-human correlations from human ratings of a different model family on the same three datasets: if the per-density weights that best track human judgment differ materially from the reported values, or if ranking models by the fixed DA Score disagrees with ranking by direct human preference on a held-out language, the central evaluation claim fails.","tokens_in":1808,"feed_emoji":"🖼️","tokens_out":2007,"duration_ms":94694,"temperature":0.7,"pith_summary":"This paper tries to establish three things about vision-language translation (VLT), the task of reading text inside images and translating it with visual context: the usual datasets are unreliable, the usual way of averaging automatic metrics misreports model quality, and fine-tuning on one high-resource language pair damages cross-lingual ability. To fix the data problem the authors build AibTrans, a parallel human-verified dataset with corrected OCR and culturally grounded translations. To fix the metric problem they propose the Density-Aware (DA) Score, which reweights BLEU, CHRF++, BERTScore, and COMET according to how well each correlates with human ratings under low, medium, and high information density. Their central adaptation claim is that a balanced multilingual sample of about 1,000 examples per direction is enough to adapt a vision-language model to VLT without eroding its multilingual generalization. If right, small curated multilingual data and density-aware scoring would make VLT benchmarking fairer and cheaper.","feed_headline":"Balanced multilingual tuning stops cross-lingual translation loss","feed_subtitle":"Fine-tuning on one high-resource pair worsens other languages; 1,000 samples per direction preserve them.","key_machinery":"Three mechanisms carry the argument. First, AibTrans: a parallel corpus of human-verified, OCR-corrected translation instances (the benchmark split is reported as 100 images and 6,993 sentence-level instances, covering Chinese into English, German, Spanish, Arabic, Russian, Japanese, and Hindi), built by manually correcting PaddleOCR output and having professional translators ground each translation in the image. Second, the information-density taxonomy: samples are clustered by bounding-box count and source token length into low, medium, and high density, and metric-human Pearson correlations are computed within each tier. Third, the Density-Aware Score, $$\\text{DA Score} = \\$\\alpha$\\cdot\\text{BLEU} + \\$\\beta$\\cdot\\text{CHRF++} + \\$\\lambda$\\cdot\\text{BERTScore} + \\phi\\cdot\\text{COMET},$$ with weights set to those per-tier correlations, replacing the naive arithmetic mean. The balanced multilingual fine-tuning strategy—sampling 500–1,000 examples per direction from MIT-10M across 8 to 13 translation directions and applying LoRA—is the mechanism that preserves multilingual competence while adapting to the task.","core_discovery":"The central discovery is that cross-lingual generalization in large vision-language models is fragile under naive task adaptation: supervised fine-tuning on one high-resource pair, English-to-Chinese, consistently degrades performance across high- and low-resource directions, while sampling 500–1,000 examples per direction from a balanced multilingual corpus improves task alignment and overall scores, with diminishing returns beyond 1,000 examples. The same study finds that current VLT benchmarks are themselves unreliable, because machine-generated references are semantically and culturally shallow and because OCR errors propagate into reference translations; AibTrans is introduced as a corrected, human-verified parallel testbed. On evaluation, no single automatic metric agrees with human judgment across all input densities: BLEU is best on short low-density text, CHRF++ and COMET on denser text, and BERTScore is unstable at medium density. The DA Score therefore reweights the four metrics by density tier and, the paper reports, tracks human scores better than their arithmetic mean. Under this protocol, cascaded models with strong multilingual LLMs are competitive with end-to-end LVLMs, and the open-source Qwen2.5-VL-7B approaches commercial performance.","pith_inferences":["The DA Score weights come from human ratings of GPT-4o outputs; a natural extension is to recompute them per model family, since models with different error patterns may need different metric weightings.","AibTrans's OCR-error statistics (90% of images with at least one correction) suggest that automatic OCR post-correction could unlock much larger corrected VLT training corpora than manual annotation alone.","The reported medium-density performance dip implies VLT difficulty is non-monotonic in input length; benchmarks that report only one average will miss the failure zone where models lack both context and brevity.","The balanced-multilingual-tuning result may transfer to other multilingual multimodal tasks such as document understanding or multilingual visual question answering, where single-language fine-tuning would similarly skew learned representations."],"forward_implications":["Fine-tuning an LVLM on a single high-resource language pair (English–Chinese) is actively harmful to other directions; a balanced multilingual sample of about 1,000 examples per direction instead improves average performance.","The DA Score, not the arithmetic mean of BLEU, BERTScore, and COMET, should be used when comparing VLT models, because simple averaging underrepresents low-quality translations and hides the medium-density failure zone.","Performance reported as one averaged number across languages can mask weak low-resource performance, so language-resource-tier reporting is needed in multilingual VLT evaluation.","Strict, heavily constrained prompts can degrade reasoning-based VLT models, while collaborative instructions and anti-hallucination constraints improve them.","External OCR remains necessary for most open LVLMs; Qwen2.5-VL-7B is the exception that performs near its OCR-supplemented ceiling without auxiliary OCR input."],"supporting_citations":[{"why":"MIT-10M is the large parallel VLT corpus whose reference and OCR quality the paper critiques, and whose translation directions supply the balanced fine-tuning samples.","marker":"[20]"},{"why":"OCRMT30K is the high-resource Chinese–English dataset used for LoRA fine-tuning and evaluation; its references are shown to be semantically and culturally shallow.","marker":"[18]"},{"why":"MTIT6 (AnyTrans) is the third evaluation dataset in the benchmark and in the reference-quality and metric-human correlation analyses.","marker":"[31]"},{"why":"Qwen2.5-VL is the strongest open-source end-to-end model studied and the main LVLM family used in fine-tuning experiments.","marker":"[6]"},{"why":"InternVL2.5 is the second LVLM family carried through the architecture comparison, OCR-dependency check, and LoRA fine-tuning.","marker":"[7]"},{"why":"LLaVA-OneVision is an open-source LVLM in the benchmark and fine-tuning tables; its strong reliance on external OCR motivates the OCR-dependency conclusions.","marker":"[19]"},{"why":"PaddleOCR provides the initial text detection for AibTrans and in cascaded pipelines; its error rate motivates the OCR-correction stage.","marker":"[30]"},{"why":"GPT-4o is the commercial model whose human-rated outputs define the metric-human correlations that the DA Score weights are derived from.","marker":"[26]"},{"why":"DeepSeek-V3 is one of the two reference-quality judges used to score semantic adequacy and cultural fit of dataset references.","marker":"[23]"},{"why":"Qwen-Max is the other reference-quality judge used in the dataset evaluation.","marker":"[8]"}],"fun_headline_variants":["Balanced tuning preserves cross-lingual skills in LVLMs","1,000 samples per language pair stops translation loss","New benchmark reveals fragile multilingual vision translation","Density-aware score beats BLEU for image text translation","Qwen2.5-VL-7B nears commercial models on multilingual VLT"],"cache_read_input_tokens":23552,"weakest_assumption_plain":"The load-bearing premise is that the human-rating correlations measured on GPT-4o outputs on MIT-10M, OCRMT30K, and MTIT6 transfer to every other model and language, and that the density categories derived from K-means are valid; if those weights do not generalize, the DA Score and the fine-tuning conclusions drawn with it are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Balanced tuning preserves cross-lingual skills in LVLMs","1,000 samples per language pair stops translation loss","New benchmark reveals fragile multilingual vision translation","Density-aware score beats BLEU for image text translation","Qwen2.5-VL-7B nears commercial models on multilingual VLT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000185,"raw_usage":{"total_tokens":1366,"prompt_tokens":1037,"completion_tokens":329,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":243}},"tokens_in":653,"tokens_out":329,"duration_ms":3746,"temperature":1.0,"reasoning_tokens":243,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:34.674559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the metric-human correlations from human ratings of a different model family on the same three datasets: if the per-density weights that best track human judgment differ materially from the reported values, or if ranking models by the fixed DA Score disagrees with ranking by direct human preference on a held-out language, the central evaluation claim fails.","supporting_citations":[{"cited_title":"MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation","cited_arxiv_id":"2412.07147","evidence_quote":"MIT-10M is the large parallel VLT corpus whose reference and OCR quality the paper critiques, and whose translation directions supply the balanced fine-tuning samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PaddleOCR provides the initial text detection for AibTrans and in cascaded pipelines; its error rate motivates the OCR-correction stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o is the commercial model whose human-rated outputs define the metric-human correlations that the DA Score weights are derived from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Qwen-Max is the other reference-quality judge used in the dataset evaluation."}],"review_version":1}