{"id":"c1fff60b-c35d-490e-8eb3-953a303b38cb","arxiv_id":"2412.07147","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MIT-10M is a 10M-pair, 14-language real-world image translation corpus with GPT-4-generated references, claimed to improve and evaluate image translation models.","lead":"MIT-10M is a new corpus of over 10 million image-text pairs in 14 languages, built from real e-commerce images using automated OCR and machine translation. The authors claim it is the largest real-world image translation dataset and show that fine-tuning a vision-language model on it substantially improves translation scores.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning gains are measured against GPT-4-generated references, so they may reflect imitation of GPT-4 rather than real translation quality; the 10-batch human check is too small to rule this out.","rationale":"The reader's verdict is CONDITIONAL, and the load-bearing concern I identify is the same one the reader flagged: reference quality. I agree that this is the weakest link in the central claim. The paper's fine-tuning result is the strongest evidence for the dataset's value, but it is measured against references generated by GPT-4 and filtered only by agreement with Google Translate. A model fine-tuned on GPT-4 translations will naturally move closer to GPT-4 style, inflating BLEU/chrF++/METEOR against those same references without necessarily improving human-judged translation quality. The 10-batch human evaluation is too small and too vaguely described to establish reference correctness at scale. This is not a dispute with external consensus; it is an internal validity question about whether the reported gains support the claim. The abstract's percentage improvements also do not match the values in Figure 6, which further weakens the quantitative headline. A human-reference recomputation on a held-out sample would settle whether the concern lands. If the gains persist on human references, the central claim is substantially supported; if they shrink or vanish, the paper's main evidence would need to be reframed. Because the dataset itself and the direction of improvement may still be valid, CONDITIONAL remains the appropriate verdict; my analysis does not move it.","tokens_in":18905,"tokens_out":6130,"duration_ms":64786,"concrete_test":"Take a random sample of 500 MIT-10M test images stratified by language and difficulty. Have professional translators, blind to the GPT-4 and Google Translate outputs, translate the text visible in each image. Compute BLEU, chrF++ and METEOR for the base Qwen2-VL and the FT@100% model against these human references, and also compute the same metrics using MIT-10M's GPT-4 references. If the fine-tuned model's advantage over the base model largely disappears on the human references — or if the GPT-4 references agree with human references at BLEU below roughly 60 — the reported gains are an artifact of matching GPT-4 style rather than evidence of translation quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim — that fine-tuning Qwen2-VL on MIT-10M raises BLEU, chrF++ and METEOR by 230%, 88% and 130% — depends entirely on the quality of MIT-10M's target-language references. These references are produced by GPT-4 and retained only when spaCy cosine similarity to Google Translate exceeds 0.8 (Section 3.3); human evaluation covers only 10 of 10,000 batches, and the protocol is not described (what exactly is 'translation accuracy'? per item or per batch? how many sentences were judged?). The fine-tuned model is then evaluated on the same MIT-10M test set, whose references come from the same GPT-4 pipeline. This makes the benchmark circular: a model trained to imitate GPT-4 will score higher against GPT-4 references than a base model, even if both produce worse translations than a human would. The manuscript's own Limitations section concedes that 'the inherent complexity of multilingual data may lead to inaccuracies that could affect the reliability of the dataset.' Additionally, the abstract's 230%/88%/130% increases do not match Figure 6, which shows BLEU 14.6→35.9 (+146%), chrF++ 29.0→56.6 (+95%), and METEOR 21.2→52.0 (+145%); the headline numbers are not reproducible from the reported data. Until reference quality is validated against independent human translations, the fine-tuning gains cannot be interpreted as evidence of improved image translation quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MIT-10M, a claimed large-scale real-world multilingual image translation dataset with 840,855 images and 10,931,115 image-text pairs across 14 languages, 28 categories, and three difficulty levels. The construction pipeline crawls images from multilingual e-commerce and general websites, applies OCR and language identification, produces translations with GPT-4, and retains translations whose spaCy-based semantic similarity to Google Translate exceeds 0.8. The authors evaluate seven image translation models on the resulting test set and fine-tune Qwen2-VL on the training set, reporting large BLEU, chrF++, and METEOR gains, as well as a fine-tuning comparison against the IIMT dataset.","tokens_in":19217,"tokens_out":6204,"duration_ms":65773,"significance":"If the dataset quality claims withstand scrutiny, MIT-10M would be a valuable community resource: it is an order of magnitude larger than existing real-world image translation datasets, it is publicly released, and it provides multi-resolution images, per-category metadata, and difficulty annotations. The paper also ships a broad comparison of current MLLMs on a realistic, multilingual image translation benchmark. However, the central quantitative claims currently rest on self-referential reference labels and on evaluation evidence that is too thin to support the headline numbers, so the contribution needs substantial additional validation before the dataset can be recommended for benchmarking.","major_comments":[{"comment":"The reference translations used for both training and evaluation are produced by GPT-4 and retained only when spaCy cosine similarity to Google Translate exceeds 0.8. Because the test references come from the same pipeline, the reported fine-tuning gains partly measure how well a model imitates the GPT-4/Google-Translate label distribution rather than how well it translates into human-quality target text. The manuscript's own Limitations section concedes that 'the inherent complexity of multilingual data may lead to inaccuracies that could affect the reliability of the dataset.' To support the central claim, the authors should validate references against independent human translations on a held-out sample, ideally stratified by language and difficulty, and report automatic metrics computed against those human references.","section":"§3.3 and §5.4"},{"comment":"The human evaluation covers only 10 of 10,000 batches (0.1%), and the protocol is not described: it is unclear what 'translation accuracy' means, whether judgments are made per sentence or per batch, how many annotators participated, or how the 99.4% figure is computed. No confidence interval or inter-annotator agreement is reported. A sample of this size cannot validate a 10.9M-pair corpus spanning 13 target languages, especially when the same pipeline is used to generate the evaluation references. The authors should report the complete protocol and a substantially larger, stratified human evaluation.","section":"§3.3"},{"comment":"The reported improvements are inconsistent. The abstract and introduction state that BLEU, chrF++, and METEOR increased by 230%, 88%, and 130%, respectively, but Figure 6 shows BLEU moving from 14.6 to 35.9 (+146%), chrF++ from 29.0 to 56.6 (+95%), and METEOR from 21.2 to 52.0 (+145%). The claim in the abstract that performance 'has tripled' is likewise not supported by the numbers in §5.4. The authors should correct the headline percentages in all locations and clarify whether the abstract refers to a different training configuration or metric.","section":"Abstract, §1, and §5.4/Figure 6"},{"comment":"The quantitative fine-tuning and evaluation results lack uncertainty estimates and are reported as single runs with no error bars or significance tests. This is especially problematic given the degree of label noise expected from the machine-translation pipeline and the strong language imbalance documented in §4.3 (English is about 49% of the data and Chinese about 60% of the English count) and in Table 3 (e.g., Digital Music has 44 images and Books 211). The claim that fine-tuning with MIT-10M 'tripled' performance should therefore be supported by multiple random seeds, confidence intervals, and per-language or per-difficulty breakdowns.","section":"§3.3, §4.3, and §5.4"}],"minor_comments":[{"comment":"The resolution criterion is stated inconsistently: §3.1 says images with resolution above 800x800 pixels are kept, while §4.4 says MIT-10M selects images with resolution above 1000x1000 pixels. Please clarify which threshold was actually used.","section":"§3.1 and §4.4"},{"comment":"There are several typographical errors: the §3.3 heading reads 'translationn', Figure 6 has 'BLUE' instead of 'BLEU', Figure 2 uses 'SRC_TEXT(ZN)' for Chinese, and the appendix shows 'SRC_TOEKN_LEN' instead of 'SRC_TOKEN_LEN'.","section":"Throughout"},{"comment":"The difficulty terminology is inconsistent: §4.2 defines 'Easy', 'Medium', and 'Hard', while §5.3 and Table 5 use 'Easy', 'Medium', and 'Difficult'. Please standardize.","section":"§4.2 and §5.3"},{"comment":"The JSON schema in the prompt contains 'en' twice, which is not a valid dictionary schema and may confuse readers trying to reproduce the OCR/translation prompt.","section":"Appendix A.1"},{"comment":"The sentence describing Figure 6 says 'BLEU increases from 14.6 for the base model (base@10%)', but the figure legend labels the first model 'Base'; this should be clarified to avoid ambiguity about which data subset is being referred to.","section":"§5.4"}],"recommendation":"major_revision","confidential_remarks":"The dataset release is a potentially useful contribution, and the scale claim is likely correct, but the evaluation protocol is not yet strong enough for the claims made. The circularity issue is an internal design property rather than a disagreement with consensus, and it can be addressed by independent human reference evaluation plus corrected reporting. I would not reject the paper, but I would not accept it in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a resource paper, and the resource is real. 840K real-world e-commerce images, 10.9M pairs across 14 languages, with category, difficulty, and resolution splits, publicly released — that's a real step up from ECOIT, DoTA, and OCRMT30K. The construction pipeline is described in enough detail to reproduce, and the comparison table is useful. I'd rather work with this corpus than with synthetic data or the small existing real sets.\n\nTwo concerns, one minor and one not.\n\nMinor: careless writing errors. The abstract's 230%/88%/130% gains don't match Figure 6, which shows roughly +146%, +95%, and +145%. 'Tripled' is only true for BLEU if you cherry-pick. The Qwen2-VL and InternVL2 citations point to earlier model papers. These are fixable but shouldn't survive a round of review.\n\nMore serious: the reference quality and the circularity. All translations are GPT-4 outputs filtered by spaCy cosine similarity to Google Translate above 0.8. Human evaluation covers 10 of 10,000 batches — that's too small to establish the 99.4% accuracy claim, and the protocol isn't described. Then the fine-tuned model is evaluated on the same MIT-10M test set, whose references come from the same GPT-4 pipeline. The gains partly measure how well the fine-tuned model imitates GPT-4's style. That doesn't make the dataset worthless — it's still a diverse multilingual parallel corpus with real images — but the 'performance tripled' claim should be reframed as 'closer to GPT-4 behavior,' not 'better translation quality.' The fine-tuning comparison against IIMT is useful but only covers four language pairs; adaptability to other real datasets isn't directly shown.\n\nTo fix the paper: run a proper human evaluation on a random sample of the test set (with a described protocol, per-language and per-difficulty numbers, confidence intervals), release the evaluation script with exact metric versions, correct the numbers and citations, and ideally show fine-tuned performance on an existing dataset like DoTA or ECOIT using the same protocol.\n\nWho this is for: anyone building or evaluating image translation models, and researchers studying dataset construction and evaluation pitfalls. It deserves a serious referee — I'd send it to peer review, but with major revision expected, not acceptance as-is.","headline":"A genuinely large and useful image-translation dataset, with real but fixable evaluation problems: machine-generated references, a too-small human check, and headline numbers that don't match the tables.","tokens_in":19714,"tokens_out":3247,"would_cite":true,"duration_ms":31432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The MIT-10M dataset provides 840,855 real-world images and 10,931,115 image–text pairs in 14 languages, claimed to be the largest corpus of its kind.","keywords":["image translation","multilingual dataset","vision-language models","fine-tuning","real-world corpus","e-commerce images","OCR","machine translation evaluation"],"falsifier":"Sample a random set of MIT-10M reference translations, have professional translators rate them blind for adequacy and fluency, and compare the scores with the claimed 99.4% accuracy from the 10 human-checked batches; if accuracy is materially lower, or if the fine-tuned model's lead shrinks when evaluated against human-translated references instead of GPT-4/Google-style references, the central claim would need revision.","tokens_in":18692,"feed_emoji":"🖼️","tokens_out":8064,"duration_ms":69529,"temperature":0.7,"pith_summary":"This paper introduces MIT-10M, a multilingual image translation corpus built from 840,855 high-resolution images crawled from real e-commerce and general web pages, with 10,931,115 image–text pairs spanning 14 languages and 28 categories. The authors claim this is the largest real-world image translation dataset to date, and they support the claim with a three-stage pipeline that filters images, recognizes embedded text with GPT-4o, translates it with GPT-4, and keeps translations only when they agree with Google Translate at a spaCy cosine similarity above 0.8. The corpus includes easy, medium, and hard splits based on text length and bounding-box count, plus images at three resolutions. On the MIT-10M test set, seven state-of-the-art models score low on BLEU, chrF++, and METEOR, which the authors take as evidence that the benchmark reflects real difficulty; fine-tuning Qwen2-VL on the training set raises BLEU by 230%, chrF++ by 88%, and METEOR by 130%.","feed_headline":"Largest real-world image translation set: 840K images, 14 languages","feed_subtitle":"Fine-tuning on MIT-10M lifts Qwen2-VL BLEU by 230%, with 28 categories and easy-to-hard splits.","key_machinery":"The central object is the MIT-10M corpus and its three-stage construction pipeline. Web pages are crawled and deduplicated; EasyOCR, langid, and langdetect prefilter images that contain text; GPT-4o performs precise OCR; GPT-4 translates the recognized text into 13 languages; and Google Gemini 1.5 Pro provides a second translation whose spaCy cosine similarity must exceed 0.8 for a pair to be kept. Difficulty labels come from bounding-box count and token length, and each image is released at original, 768-pixel, and 500-pixel widths. The dataset is both the training resource and the evaluation benchmark, so the pipeline's filtering decisions determine what the reported gains measure.","core_discovery":"The paper constructs MIT-10M, a corpus of 840,855 high-resolution images crawled from real e-commerce web pages in eight source languages, paired with machine translations into 13 target languages plus the original, for a total of 10,931,115 image–text pairs across 14 languages and 28 product categories. Each sample carries metadata — bounding-box count, token length, category, difficulty level, and three image sizes — and the pipeline filters translations by asking GPT-4 and Google Translate to agree, requiring a spaCy cosine similarity above 0.8, with 10 of 10,000 batches human-checked at 99.4% accuracy. On the MIT-10M test set, seven end-to-end and cascade models score low on BLEU, chrF++, and METEOR, which the authors read as evidence that the benchmark captures real-world difficulty; fine-tuning Qwen2-VL on the MIT-10M training set raises BLEU by 230%, chrF++ by 88%, and METEOR by 130%, and a controlled comparison against IIMT at similar data size shows MIT-10M yields higher BLEU on four language pairs.","pith_inferences":["If GPT-4's style is systematically embedded in the reference labels, part of the fine-tuning gain may reflect imitation of GPT-4 rather than general translation quality; evaluating against human-curated references would separate these effects.","The 0.8 cosine-similarity cutoff between two machine translators may systematically discard hard, idiomatic, or low-resource text, so the corpus could under-represent the most difficult real-world translation cases.","The same pipeline could be applied to additional domains such as social media screenshots, signage, and documents, or extended to other multimodal generation tasks that need aligned image–text data.","The per-sample metadata could support controlled studies of how vision-language models use resolution and layout cues, since each image is released at three sizes."],"forward_implications":["With 14 languages and 28 categories, MIT-10M lets image translation models be trained and compared across a much wider language and domain spread than previous real-world datasets.","Because the test set includes easy, medium, and hard splits, models can be analyzed for how performance degrades as text length and bounding-box count grow; the paper reports average METEOR falling from 18.3 on easy to 14.0 on hard.","Fine-tuning on MIT-10M improves Qwen2-VL's BLEU from 14.6 to 35.9, chrF++ from 29.0 to 56.6, and METEOR from 21.2 to 52.0 at full training data, and even 10% of the data beats the base model.","At controlled data size, fine-tuning on MIT-10M outperforms fine-tuning on IIMT on DE-EN, EN-DE, EN-FR, and FR-EN BLEU."],"supporting_citations":[{"why":"Supplies GPT-4, the engine that translates the OCR text into 13 target languages and therefore defines the reference labels.","marker":"Achiam et al., 2023"},{"why":"Supplies Google Gemini 1.5 Pro, the independent second translation whose semantic similarity filters the references.","marker":"Gemini, 2024"},{"why":"Introduces the Qwen2-VL model family that is fine-tuned and evaluated, anchoring the reported 230% BLEU gain.","marker":"Bai et al., 2023"},{"why":"Provides IIMT, the comparison dataset used to show MIT-10M's fine-tuning advantage at controlled size.","marker":"Lan et al., 2024"},{"why":"Defines BLEU, one of the three metrics used to report both evaluation and fine-tuning results.","marker":"Papineni et al., 2002"},{"why":"Defines chrF++, the character-level metric the paper recommends for short-sentence translation evaluation.","marker":"Popović, 2017"},{"why":"Defines METEOR, the third metric used in all model comparisons.","marker":"Banerjee and Lavie, 2005"}],"fun_headline_variants":["MIT-10M: 10M pairs, 840K images, 14 languages","Fine-tuning on MIT-10M triples BLEU for image translation","Real-world image translation corpus: 10M pairs, 14 languages","MIT-10M benchmark: 28 categories, 3 difficulty levels, 14 languages","MIT-10M dataset boosts BLEU by 230% on image translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reference translations generated by GPT-4 and kept only when Google Translate's translation has spaCy cosine similarity above 0.8 are accurate enough to serve as gold labels for both training and evaluation.","fun_headline_variants_meta":{"raw":{"variants":["MIT-10M: 10M pairs, 840K images, 14 languages","Fine-tuning on MIT-10M triples BLEU for image translation","Real-world image translation corpus: 10M pairs, 14 languages","MIT-10M benchmark: 28 categories, 3 difficulty levels, 14 languages","MIT-10M dataset boosts BLEU by 230% on image translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000407,"raw_usage":{"total_tokens":2134,"prompt_tokens":983,"completion_tokens":1151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":1044}},"tokens_in":599,"tokens_out":1151,"duration_ms":11240,"temperature":1.0,"reasoning_tokens":1044,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:04:08.961337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a random set of MIT-10M reference translations, have professional translators rate them blind for adequacy and fluency, and compare the scores with the claimed 99.4% accuracy from the 10 human-checked batches; if accuracy is materially lower, or if the fine-tuned model's lead shrinks when evaluated against human-translated references instead of GPT-4/Google-style references, the central claim would need revision.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides IIMT, the comparison dataset used to show MIT-10M's fine-tuning advantage at controlled size."}],"review_version":1}