{"id":"00ac59b7-9a35-47ea-8309-e67fa9394390","arxiv_id":"2509.05146","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"PRIM provides the first real-world, multilingual in-image translation benchmark, and the proposed VisTrans end-to-end model improves visual quality over prior end-to-end systems while trailing strong cascade models on translation accuracy.","lead":"A research team built PRIM, a small benchmark of real-world photos of one-line text with translations into five languages, and a model called VisTrans that rewrites the text while preserving the photo's background. The model produces cleaner output images than earlier end-to-end systems, but its translated text is less accurate than a strong OCR-plus-translation pipeline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VisTrans's 'better translation quality' claim is unsubstantiated: Table 1 lacks significance tests and omits the recent DebackX baseline from PRIM, leaving the comparative claim unverifiable.","rationale":"The reader's weakest assumption focuses on the synthetic-to-real transfer of the training set. While that is a legitimate generalization concern, the paper already tests on real-world PRIM images, and VisTrans obtains non-trivial BLEU/FID scores, so the transfer is partly demonstrated. The more load-bearing concern is the validity of the comparative claim itself. The abstract overstates performance relative to Table 1: the strongest cascade baseline beats VisTrans by a wide margin, and VisTrans's COMET is below PEIT-Render. Additionally, the omission of DebackX from the main benchmark, despite the paper comparing against it elsewhere, leaves the comparison set incomplete. These issues directly affect whether the central claim is supported, and they are testable. The reader did note the missing DebackX and lack of error bars in the rationale, but did not make them the primary weakest assumption. My read agrees with the conditional accept: the dataset is a useful contribution, and the concerns are addressable via additional experiments and rephrasing. Thus the verdict remains UNCHANGED.","tokens_in":15598,"tokens_out":6825,"duration_ms":68357,"concrete_test":"Run DebackX on the PRIM test set using the exact same evaluation pipeline (EasyOCR -> BLEU/COMET, pytorch-fid) and add it to Table 1. Separately, perform bootstrap resampling (e.g., 10,000 resamples over the 340 images per direction) to compute 95% confidence intervals for the BLEU and COMET differences between VisTrans and PEIT-Render. If DebackX outscores VisTrans, or if the confidence interval for the PEIT-Render comparison includes zero, the headline claim should be revised to 'comparable' or 'better under specific metrics' rather than 'better.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that VisTrans achieves better translation quality and visual effect than other models. However, Table 1 shows VisTrans's average BLEU of 11.3 is far below the cascade EasyOCR-NLLB-Render (23.0), and its average COMET of 47.0 is actually lower than PEIT-Render's 48.0. The abstract claims superiority without these qualifications. No confidence intervals or significance tests are reported; with only 340 images per direction, the ~1 BLEU gap between VisTrans and PEIT-Render may be noise. More critically, DebackX, a recent end-to-end IIMT model that the authors themselves evaluate against VisTrans on IIMT30k (Table 4), is missing from the PRIM benchmark. Omitting a relevant state-of-the-art baseline from the main comparison makes the claim 'better than other models' unverifiable and potentially false if DebackX outperforms VisTrans on PRIM.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PRIM, a benchmark for practical in-image multilingual machine translation (IIMMT) consisting of real-world captured source images with one-line text, five translation directions (En-De, En-Fr, En-Cs, En-Ru, En-Ro), and 340 images per direction with two reference target images per source image. The authors also propose VisTrans, an end-to-end model that separates background information from visual-text information using a two-stage training procedure, a codebook, and a subword-to-character decoder. Experiments on PRIM compare VisTrans with pre-trained and non-pre-trained cascade systems, with prior end-to-end IIMT models, and with the golden reference. The paper claims that VisTrans achieves better translation quality and visual effect than other models. The visual-quality claim is substantially supported by the reported FID (VisTrans 28.8 vs. 69.1 for TranslatotronV and >100 for cascades), but the translation-quality claim is not: Table 1 shows VisTrans average BLEU 11.3 vs. 23.0 for EasyOCR-NLLB-Render and average COMET 47.0 vs. 48.0 for PEIT-Render.","tokens_in":15900,"tokens_out":5417,"duration_ms":59439,"significance":"If the results hold, the paper makes two valuable contributions. PRIM is, to my knowledge, the first publicly available multilingual IIMT benchmark using real-world source images, and the authors provide code and data. VisTrans is a serious end-to-end attempt at practical IIMT, and its large FID improvement over cascades and TranslatotronV is a meaningful result, as is its demonstrated ability to handle multi-line synthetic IIMT30k data. The paper also provides a large synthetic training set with multilingual directions. However, the headline claim of better translation quality is not supported by the paper's own table, and the absence of the recent DebackX baseline from the main PRIM comparison, together with the lack of significance testing, leaves the comparative claim under-supported. The synthetic-to-real transfer of the training procedure is also not validated. The central empirical claims are partially supported and the issues are fixable, but the paper needs revision before the claims can be accepted as stated.","major_comments":[{"comment":"The claim that VisTrans achieves \"better translation quality ... compared to other models\" is contradicted by Table 1. VisTrans has average BLEU 11.3 vs. 23.0 for EasyOCR-NLLB-Render and average COMET 47.0 vs. 48.0 for PEIT-Render and 62.7 for EasyOCR-NLLB-Render. VisTrans is the best end-to-end model in the table, but it is not better in translation quality than all models. Please revise the abstract and conclusion to state the qualified finding, e.g., \"best visual quality and competitive translation quality among end-to-end models,\" or report the full comparison without the blanket superiority claim.","section":"Abstract; §7 (Conclusion)"},{"comment":"DebackX is identified as a recent end-to-end IIMT model in Figure 1 and is compared against VisTrans on IIMT30k in Table 4, but it is absent from the PRIM evaluation in Table 1. Since the paper's central comparative claim is that VisTrans is better than other models, omitting the strongest recent end-to-end IIMT baseline from the main benchmark makes that claim unverifiable. Please add DebackX to the PRIM evaluation, or clearly justify its exclusion and relax the comparative claim accordingly.","section":"Table 1 / §5.3"},{"comment":"No confidence intervals, standard deviations, or significance tests are reported for any metric. With only 340 images per direction, the BLEU difference between VisTrans (11.3) and PEIT-Render (10.4) is about 0.9, and the COMET difference is -1.0 (47.0 vs. 48.0); these differences are plausibly within noise. Report bootstrap confidence intervals, paired significance tests, or per-direction variance to support the relative ranking of systems.","section":"Table 1 / §5.1"},{"comment":"The training set is entirely synthetic: source text is rendered with TRDG, target text uses Arial, and backgrounds are extracted from video frames. PRIM, by contrast, contains real-world captured advertising and subtitle images. No experiment measures this simulation-to-real transfer: there is no training-on-real-data comparison, no per-domain breakdown, and no analysis of how the domain gap affects BLEU/COMET/FID. The \"practical\" claim therefore rests on an unvalidated transfer assumption. The Limitations section acknowledges computational cost and codebook simplicity but not this risk, which should be addressed empirically or explicitly listed as a limitation.","section":"§3 Training Set"}],"minor_comments":[{"comment":"Several language errors should be fixed: \"convince the lack\" should be \"compensate for the lack\" or \"address the lack\"; in §7 \"retrains\" should be \"retains\"; and \"5translation\" is missing a space.","section":"Abstract; §3; §7"},{"comment":"The table formatting is poor in places, e.g., \"10.4 /45.114.0\" and \"12.8 /53.59.9 / 46.8\". Please ensure values and slashes are clearly separated.","section":"Table 1"},{"comment":"The caption uses \"ML.\" and \"RB.\" without defining them. Spell out \"multilingual\" and \"real-world background\" in the caption or legend.","section":"Figure 1"},{"comment":"The \"Speedup\" column in Table 3 is not defined precisely. State what is measured (e.g., inference-only time, batch size, hardware) and whether the \"None\" row means removing the S2C decoder entirely and generating codes directly from the 1-Pass representation.","section":"§6.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset and model are potentially solid contributions, and the FID result is striking. My main concern is that the abstract and conclusion overclaim translation quality relative to Table 1, and that the comparison is incomplete without DebackX on PRIM and significance tests. If the authors re-frame the claims and add the missing baseline/statistics, the paper could be acceptable. The journal should also consider whether the synthetic-to-real transfer limitation is adequately handled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it delivers a real contribution: PRIM, the first public real-world multilingual benchmark for in-image machine translation, with 340 images per direction across 5 directions and two reference translations each. The construction is careful—manual inpainting, human-level translation quality checks, and a sensible source selection from video subtitles and e-commerce ads. Second, the paper's headline claim is not supported by its own Table 1. VisTrans reaches 11.3 average BLEU, but the EasyOCR-NLLB-Render cascade gets 23.0, and VisTrans's COMET (47.0) is actually below PEIT-Render (48.0). The abstract says \"better translation quality and visual effect\" without qualification, which is misleading.\n\nWhat the paper does well: the visual quality result is strong. VisTrans's FID of 28.8 crushes TranslatotronV (69.1) and all cascades (100+), and the background preservation is visible in the case study. The architecture—separate encoding of background and text, a two-stage training with auxiliary reconstruction tasks, and a subword-to-char decoder to bridge the codebook granularity—is a plausible step forward for end-to-end IIMT. The robustness test on IIMT30k, where VisTrans beats DebackX by 4-6 BLEU, suggests the approach generalizes to multi-line text.\n\nSoft spots, in proportion. The translation-quality overclaim is the most visible issue; the paper's own Section 5.3 already walks it back to \"except for pre-trained models, our VisTrans and PEIT-Render achieve better performance.\" That's a fixable wording problem. More substantive: there are no error bars or significance tests. With 340 images per direction, the 0.9 BLEU edge over PEIT-Render is within noise. The missing DebackX baseline from the PRIM table is the real gap. The authors have DebackX numbers on IIMT30k, so there's no excuse not to run it on PRIM. Without that, the \"better than other models\" claim is unverifiable. The sim-to-real transfer is also unablated—training is entirely synthetic with Arial target fonts—so the strong FID result on real PRIM images is promising but not fully explained. These are all addressable in revision.\n\nThe central visual-quality argument holds. The translation-quality argument is overstated but the model is genuinely competitive among end-to-end systems and far ahead of TranslatotronV. This paper is for researchers in IIMT and image-to-image generation; the benchmark alone is worth citing even if the model has limitations.\n\nRecommendation: this deserves a serious referee. I'd like to see the missing baseline added and the claims rebalanced before acceptance.","headline":"PRIM is a genuinely useful new real-world IIMT benchmark and VisTrans has credible visual-quality gains, but the abstract overclaims translation quality and the missing DebackX baseline must be addressed before the comparative claims are verifiable.","tokens_in":16337,"tokens_out":4163,"would_cite":true,"duration_ms":42119,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces PRIM, a real-world multilingual in-image translation benchmark, and VisTrans, an end-to-end model that separates text and background to translate photos without damaging the scene.","keywords":["in-image machine translation","multilingual translation","image-to-image translation","real-world benchmark","visual text generation","codebook quantization","two-stage training","text image translation"],"falsifier":"Train or fine-tune VisTrans on PRIM's real source images paired with their annotated target images, then compare BLEU and FID against the synthetic-trained model; if the real-trained model does not improve, or if synthetic-trained performance collapses on held-out fonts, backgrounds, and text lengths outside the training renderer, the transferability premise is falsified. A second check: measure background preservation directly by computing FID on background-only regions to confirm that adding Hback preserves the scene rather than regenerating it.","tokens_in":15560,"feed_emoji":"🖼️","tokens_out":5834,"duration_ms":60754,"temperature":0.7,"pith_summary":"This paper argues that in-image machine translation can move from simplified synthetic setups to practical, real-world conditions. To make that case, it introduces PRIM, a new benchmark of real-world photographed one-line text images with complex backgrounds, varied fonts, diverse text positions, and five English-to-X translation directions. It also proposes VisTrans, an end-to-end model that processes visual text and background information separately, then combines them when generating the translated image. The authors report that VisTrans outperforms earlier end-to-end and most cascade baselines on both translation quality and visual fidelity, while a strong pretrained cascade still leads on translation quality.","feed_headline":"VisTrans translates text inside real photos, keeping backgrounds intact","feed_subtitle":"New PRIM benchmark covers five languages; end-to-end model beats prior systems on BLEU and FID.","key_machinery":"The central mechanism is VisTrans's separation of visual text and background into two representation streams. The text stream runs the source image through an MT Encoder, generates subword target text with a 1-Pass Decoder using a language tag, converts it to character-level representations via a Subword-to-Char (S2C) decoder, and predicts codebook codes with a 2-Pass Decoder; the background stream produces a background representation that is added to the text-code representation before image decoding. The codebook stores visual text patches, and the S2C decoder bridges the granularity gap between subwords and image codes. Two-stage training with auxiliary tasks—background reconstruction, te","core_discovery":"VisTrans is an end-to-end model that translates text inside a photo while preserving the surrounding scene. It encodes the source image through separate vision-transformer streams: one captures the visual appearance of the text, the other captures the background. A codebook quantizes target text-image patches, and the model generates the target image by autoregressively predicting codebook codes from the translated text, then adding the preserved background. The model is trained in two stages, first teaching these branches to reconstruct text-images and backgrounds, then teaching the translation-to-code path. On PRIM, VisTrans reports an average BLEU of 11.3 and FID of 28.8, outperforming th","pith_inferences":["The two-stage recipe—translate to text first, then render text into the image—could transfer beyond one-line IIMT to document image translation, signboard translation, or any text-bearing image where background preservation matters.","If the synthetic-to-real transfer is confirmed, the training pipeline can be scaled cheaply to many more languages and scripts by rendering parallel text from existing corpora, without new human annotation.","The S2C decoder ablation suggests subword-to-patch granularity mismatch is a central bottleneck, so finer intermediate representations such as characters, strokes, or adaptive group sizes are a natural next lever for quality.","PRIM's two-reference design (GPT-4 and Google Translate) lets future work separate translation accuracy from rendering fidelity, for example by scoring OCR round-trip agreement between the two references."],"forward_implications":["If VisTrans's design holds, end-to-end in-image translation no longer requires the OCR-translate-render cascade, avoiding the background damage and incomplete-text-rendering failure modes the paper documents.","Multilingual support is achieved without separate models per language pair: a language tag steers the same decoder across five translation directions, suggesting easy extension to more languages.","Because the codebook stores visual text patches, the model can re-render translated text at a fitting font size, avoiding the truncation that limits cascade models on long text.","Reported results on the IIMT30k dataset indicate the architecture transfers to multi-line and more complex text layouts when trained appropriately, not just PRIM's one-line images.","PRIM provides a reusable real-world benchmark with two reference translations per direction, making future IIMT systems comparable on both translation quality and visual fidelity."],"supporting_citations":[{"why":"Defines the end-to-end in-image machine translation task that PRIM and VisTrans extend.","marker":"(Mansimov et al., 2020)"},{"why":"TranslatotronV, the end-to-end baseline VisTrans compares against and improves on.","marker":"(Lan et al., 2024)"},{"why":"Supplies real-world video-subtitle images used as PRIM source images.","marker":"(Ma et al., 2024)"},{"why":"Supplies real-world web and e-commerce advertising-board images used as PRIM source images.","marker":"(Li et al., 2025a)"},{"why":"Provides the CVSS design paradigm of real source input with synthetic target output that PRIM's annotation follows.","marker":"(Jia et al., 2022b)"},{"why":"Provides the MTed parallel texts used to render the synthetic training set.","marker":"(Duh, 2018)"},{"why":"Justifies synthesizing large-scale training images with varied fonts and backgrounds to approximate real scenes.","marker":"(Jaderberg et al., 2014)"},{"why":"Its NLLB model powers the strongest cascade baseline, the main translation-quality rival.","marker":"(NLLB Team et al., 2022)"},{"why":"PEIT is the strongest cascade-style baseline VisTrans is compared with, and a source of single-line image conventions.","marker":"(Zhu et al., 2023)"},{"why":"AnyTrans is the advanced cascade baseline whose failure on long text motivates VisTrans's rendering approach.","marker":"(Qian et al., 2024)"}],"fun_headline_variants":["VisTrans translates in-photo text, preserving the scene","PRIM benchmark: real-world images for in-image multilingual translation","End-to-end model translates text in complex photos, beats baselines","New model handles multilingual text on real photos with complex backgrounds","In-image translation steps out of synthetic data with PRIM dataset"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that a model trained on synthetically rendered text images—Arial target fonts on video-frame backgrounds—will transfer to the real-world photographs in PRIM; if that simulation-to-real gap is large, the reported quality and multilingual gains will not survive deployment.","fun_headline_variants_meta":{"raw":{"variants":["VisTrans translates in-photo text, preserving the scene","PRIM benchmark: real-world images for in-image multilingual translation","End-to-end model translates text in complex photos, beats baselines","New model handles multilingual text on real photos with complex backgrounds","In-image translation steps out of synthetic data with PRIM dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1173,"prompt_tokens":720,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":464,"tokens_out":453,"duration_ms":5239,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:32:26.040833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or fine-tune VisTrans on PRIM's real source images paired with their annotated target images, then compare BLEU and FID against the synthetic-trained model; if the real-trained model does not improve, or if synthetic-trained performance collapses on held-out fonts, backgrounds, and text lengths outside the training renderer, the transferability premise is falsified. A second check: measure background preservation directly by computing FID on background-only regions to confirm that adding Hback preserves the scene rather than regenerating it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the end-to-end in-image machine translation task that PRIM and VisTrans extend."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TranslatotronV, the end-to-end baseline VisTrans compares against and improves on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MTed parallel texts used to render the synthetic training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PEIT is the strongest cascade-style baseline VisTrans is compared with, and a source of single-line image conventions."}],"review_version":1}