{"id":"a9f67739-dc5d-4307-b3ea-4b29216a8e3a","arxiv_id":"2412.12627","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A framework that generates images from source sentences with a Stable Diffusion model fine-tuned by a scene-graph reward, then feeds them into a multimodal LLM, is claimed to improve machine translation, but key comparisons are confounded.","lead":"Researchers built a translation system that first uses Stable Diffusion to generate a picture of what the source sentence describes, then feeds both text and picture into a multimodal language model to produce the translation. They report large BLEU gains on the Multi30K benchmark, though the headline improvement figure conflicts with the paper's own tables and the WMT comparisons may be confounded by fine-tuning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WMT24 gains are confounded by the absence of a fine-tuned text-only baseline; the claim that visual information improves general-domain MT is not yet established.","rationale":"The central claim has two legs: (1) Multi30K gains and (2) WMT24 general-domain gains. Leg (1) is reasonably supported by Table 3's ✓✗ row, which appears to be a text-only fine-tuned baseline (27.2 → 32.9) versus full IMAGE (45.3); the gain is large even if the abstract's '>14' is an overstatement (the paper computes 13.7 in §5.1). Leg (2) is the riskier leg: Table 2 compares IMAGE only to zero-shot LLMs, so the improvement is confounded by fine-tuning. Since the paper explicitly frames WMT24 results as evidence that 'visual information enhances the translation ability of LLMs in the general MT task,' the absence of a matched text-only baseline is the weakest link. A single experiment—fine-tuning the same backbone on the same Multi30K data without images and evaluating on WMT24—would settle this attribution. The reader's weakest_assumption identifies exactly this issue, so I agree with the conditional verdict; the concern is concrete and testable but does not by itself justify rejection without running the control.","tokens_in":111,"tokens_out":7758,"duration_ms":77263,"concrete_test":"Fine-tune Vicuna-7B (or the same DreamLLM backbone without vision) on the Multi30K training set (En-De and En-Fr) using only the text-to-text LM loss (no image tokens, no L_IMAGERL), with identical hyperparameters (1.5 epochs, batch size 16, peak LR 2e-5, 3% warmup). Evaluate this baseline on the WMT24 test sets (En->De, En->Zh, En->Hi, En->Cs) using the same BLEU and COMET metrics as Table 2. If this text-only fine-tuned baseline comes within ~1 BLEU of IMAGE's scores, the visual-signal attribution for general MT is unsupported; if IMAGE still outperforms by a clear margin, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's claim that generated images improve general-domain machine translation (Section 5.2, Table 2) lacks a matched control. IMAGE is fine-tuned on Multi30K (both the LLM and the diffusion model), while the Vicuna-7B comparator in Table 2 is presented as an off-the-shelf model; the footnote exempts only Tower/ALMA from fine-tuning, and Section 4.2 describes Vicuna as a general instruction-following model. Thus the +3.9 BLEU and +8.2 COMET gains on WMT24 could stem entirely from additional translation fine-tuning on Multi30K rather than from the generated visual signal. Table 3 demonstrates that fine-tuning alone (L_MLLM without RL) raises Multi30K BLEU from 27.2 to 32.9, so a similar improvement likely transfers to WMT24. Without a text-only Vicuna-7B fine-tuned on the identical Multi30K data and evaluated on WMT24, the attribution of the WMT24 gains to the images is confounded. This is the load-bearing weak point in the broader claim that visual imagination improves text-only MT in general domains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IMAGE, an end-to-end framework that first generates an image conditioned on the source sentence using a Stable Diffusion model fine-tuned with a scene-graph consistency reward, and then feeds the generated image together with the source text into a multimodal large language model (DreamLLM/Vicuna-7B) for translation. The framework is evaluated on Multi30K (English-German, English-French) and WMT24 (four language pairs) against traditional multimodal MT models, text-only LLMs, and the DreamLLM baseline. The central claims are that explicitly generated visual information improves both multimodal and text-only machine translation and that the proposed automated reward eliminates the need for manual image-text annotations, with an average improvement of more than 14 BLEU points on Multi30K.","tokens_in":18973,"tokens_out":11363,"duration_ms":84651,"significance":"If the claims are supported, the contribution is valuable: the method extends image-free multimodal MT beyond inference-time hallucination to explicit generation, uses a reward that avoids manual image annotations, and reports gains across several language pairs with ablations for the loss terms and modules. The paper also provides an image-quality analysis via CLIPScore and qualitative examples. The main strengths are the clarity of the proposed pipeline and the breadth of the experiments. However, the strength of the contribution depends on resolving an ambiguity in the text-only baselines and on correcting the headline BLEU figure, which currently overstates the reported average.","major_comments":[{"comment":"The status of the text-only baselines is ambiguous, and this ambiguity is load-bearing for the WMT24 claims. Section 5.1 says the comparison is among \"methods fine-tuned on the same training set,\" and Table 3's (✓ ✗) row exactly matches the Vicuna-7B row of Table 1, which suggests that the Vicuna-7B baseline was fine-tuned with L_MLLM on Multi30K. However, Section 4.2 introduces Vicuna-7B as a general instruction-following model, and Table 2's footnote only exempts Tower-7B and ALMA from fine-tuning on the WMT24 test set. If the Vicuna-7B numbers in Table 2 are zero-shot, then the reported +3.9 BLEU and +8.2 COMET gains on WMT24 are confounded with the benefit of Multi30K fine-tuning, since Table 3 shows that L_MLLM alone raises Multi30K BLEU from 27.2 to 32.9. Please state explicitly for every baseline whether and on which data it was fine-tuned; if the Table 2 Vicuna row is not the Multi30K-fine-tuned text-only model, add that control and re-evaluate the claim that visual information improves general-domain MT.","section":"§4.2, §5.1–5.2, Tables 1–3"},{"comment":"The abstract states \"an average improvement of more than 14 BLEU points\" on Multi30K and MSCOCO, and the Introduction repeats this claim, but the calculation in Section 5.1 is (12.4+10.6+11.4+21.1+16.7+10)/6 = 13.7 BLEU, which is not \"more than 14.\" The headline number should be corrected to \"13.7 BLEU\" or \"approximately 13.7 BLEU,\" and any other occurrences should be updated for consistency.","section":"Abstract and §5.1"},{"comment":"The method is repeatedly described as \"human feedback\" and \"reinforcement learning from human feedback,\" but the reward in Eq. (8) is computed entirely from off-the-shelf scene-graph parsers and a sentence-transformer similarity model; no human annotations or preference judgments are collected. This naming overstates the role of human feedback and should be replaced with a neutral description such as \"automated scene-graph consistency reward.\" In addition, the text says the reward ranges from 0 to 1, but the cosine similarity used in Eq. (7) can be negative; please clarify whether the similarity values are clamped or otherwise normalized to that range.","section":"§3.3, Eq. (6)–(8)"},{"comment":"The tables report that bolded results are statistically significant with p<0.01 in a paired t-test, but no information is given about the number of independent runs, random seeds, variance, or the test-set sizes for the WMT24 language pairs. Without this information, the significance claim and the assertion of \"highest statistically significant scores\" cannot be verified. Please report the number of evaluation examples per test set, the number of runs or seeds, and either error bars or confidence intervals.","section":"§4.3, §5.1, §5.2, Tables 1–2"}],"minor_comments":[{"comment":"\"multimodel\" should be \"multimodal\" in the phrase \"advancing the multimodel MT.\"","section":"Abstract and Introduction"},{"comment":"\"inculding\" should be \"including.\"","section":"§2.2"},{"comment":"The sentence about the w/o VS ablation appears to have the numbers reversed: it says \"(45.43/38.6/37.5 without CLIP, compared to 39.2/35.1/33.2 with CLIP),\" but Table 4 shows 45.3/38.6/37.5 for IMAGE with the vision encoder and 39.2/35.1/33.2 for w/o VS; please correct the wording.","section":"§5.5, Table 4"},{"comment":"The reported \"average improvement\" for the low-resource tasks should state the baseline set over which the average is computed, since the table contains several baselines and the average is not uniquely defined otherwise.","section":"§5.2"},{"comment":"\"Eglish\" should be \"English.\"","section":"Appendix A.1"},{"comment":"The phrase \"Ambiguous COCO\" is used but not defined; if it refers to the Multi30K MSCOCO split, please say so explicitly to avoid confusion.","section":"§5.6, Table 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has one real result and one inflated wrapper. The real result is on Multi30K, where generating an image with a scene-graph-RL-fine-tuned Stable Diffusion and feeding it into an MLLM gives a small but consistent BLEU gain over a text-only fine-tuned control (roughly 1.7-2.4 BLEU across the three En→De test sets, per Table 4). The wrapper is the abstract's claim of 'more than 14 BLEU' and the WMT24 general-domain gains, both of which come from comparing against off-the-shelf baselines rather than matched fine-tuned text-only models.\n\nWhat's actually new: the specific combination—diffusion model fine-tuned with a scene-graph consistency reward, then that image used as input to an MLLM translator—is new, as far as I know. Prior imagination-based MT (Elliott and Kádár, ImagiT) didn't use diffusion or RL. The ablation in Table 4 is informative: removing the generated image drops BLEU, and removing the vision encoder drops it more. Table 5 shows the RL fine-tuning genuinely improves image-text CLIP scores. That is a credible mechanism, and it works without any image annotations. Credit is due for those pieces.\n\nSoft spots, in order of importance:\n\n1. WMT24 claims are confounded. The comparison in Table 2 is IMAGE (fine-tuned on Multi30K) vs. off-the-shelf Vicuna/LLaMA/Alpaca. Table 3 shows text-only fine-tuning with L_MLLM raises Multi30K BLEU from 27.2 to 32.9, so a similar boost likely transfers to WMT24. Without a text-only Vicuna fine-tuned on the same Multi30K data and evaluated on WMT24, you cannot attribute the +3.9 BLEU on En→Zh or +12.9 on En→Cs to visual information. This is the load-bearing weakness for the paper's headline general-domain conclusion.\n\n2. The '14 BLEU' claim is not supported by the paper's own numbers. Section 5.1 computes 13.7 BLEU average over Vicuna; using the table values I get 13.85. Either way it's under 14. The abstract should be corrected.\n\n3. 'Human feedback' is a misnomer. The reward is an automated scene-graph similarity computed by off-the-shelf parsers and Sentence-BERT. That's a legitimate heuristic, but calling it RLHF overstates it.\n\n4. Minor: no error bars, seeds, or test sizes; paired t-test significance claims are unverifiable. And the 'first to propose MLLM for MT' is too strong given earlier imagination-based methods, though the diffusion+RL variant is new.\n\nNet: the Multi30K result is probably real, the WMT24 general-domain result is not established, and the presentation overclaims. I'd send this to peer review—a serious referee can push for the matched text-only baseline and corrected claims. It's citable for the imagination+diffusion idea with caveats; I'd bring it to a reading group mainly to discuss the baseline problem.","headline":"A real but modest Multi30K effect wrapped in inflated claims; the WMT24 general-domain gains are confounded by missing fine-tuned text-only baselines.","tokens_in":19532,"tokens_out":7183,"would_cite":true,"duration_ms":53791,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Imagining the source sentence as an image lifts machine translation by about 14 BLEU points.","keywords":["machine translation","multimodal machine translation","Stable Diffusion","scene graph consistency reward","reinforcement learning from human feedback","multimodal large language model","visual imagination","low-resource translation"],"falsifier":"Train a text-only Vicuna-7B model on the same Multi30K training pairs with the same $L_{\\mathrm{MLLM}}$ loss and training schedule, then evaluate it on the WMT24 test sets; if its BLEU/COMET scores are close to IMAGE's, the reported general-domain gains are not caused by the generated images.","tokens_in":18513,"feed_emoji":"🖼️","tokens_out":8922,"duration_ms":67360,"temperature":0.7,"pith_summary":"This paper claims that machine translation improves when the model first generates an image of what the source sentence describes and then translates from the text plus that image. To avoid needing paired image-text training data, the image is produced by a Stable Diffusion model that is fine-tuned with a reinforcement-learning reward measuring how well the scene graph of the generated image matches the scene graph of the source sentence. The resulting framework, IMAGE, is reported to beat traditional multimodal MT systems and text-only LLM baselines on Multi30K, with an average gain of about 14 BLEU points over the Vicuna-7B text-only baseline (Section 5.1 computes 13.7), and to improve BLEU/COMET on the WMT24 general-domain test sets as well. The attraction, if the claim holds, is that visual information becomes usable for any text-only translation corpus, including low-resource pairs, without any manual image annotations.","feed_headline":"Imagining the sentence as an image lifts translation by ~14 BLEU","feed_subtitle":"No paired images are needed: a scene-graph reward makes Stable Diffusion draw what the source sentence describes.","key_machinery":"The load-bearing object is the scene-graph consistency reward $r(x_0, c)$ defined in Equation (8): for each triple in the linguistic scene graph (LSG) parsed from the source sentence, take its maximum similarity to any triple in the visual scene graph (VSG) parsed from the generated image, using an off-the-shelf text-similarity model, then average over all source triples. This reward is used in a Denoising Diffusion Policy Optimization (DDPO) loop to fine-tune Stable Diffusion via a REINFORCE-style policy-gradient estimate (Equation (10)), so the generated image becomes a clearer 'imagination' of the source sentence. The generated image is then embedded with CLIP and fed through a multimodal LLM (DreamLLM, built on Vicuna-7B) together with the source text for translation.","core_discovery":"The central claim is that an end-to-end multimodal LLM translation framework can use a text-to-image diffusion model as a translator's 'imagination': generate an image from the source sentence, and let the LLM translate with both text and image as input. The paper argues that the reason previous image-based translation approaches underperformed is that their generated images were not faithful to the source sentence, and it addresses this by fine-tuning Stable Diffusion with a reward derived from scene-graph consistency between the source sentence and the generated image. This reward makes the model 'imagine' scenes with the correct entities and relations, for example the right number of women and their action, without any image annotation during training. On Multi30K En→De and En→Fr, the paper reports that IMAGE outperforms the strongest traditional multimodal MT baselines and beats the text-only Vicuna-7B base by an average of 13.7 BLEU; on WMT24, it reports consistent improvements across four language pairs, with the largest gains on the low-resource En→Cs and En→Hi directions.","pith_inferences":["A reader should not take the WMT24 numbers as isolating the visual signal: the paper compares against text-only LLMs that were not fine-tuned on the same Multi30K data, leaving the gain from additional fine-tuning unseparated.","One testable extension is to run the same fine-tuning recipe with the $L_{\\mathrm{MLLM}}$ loss only (no images, no RL) on WMT24; if that text-only model closes most of the gap, the visual contribution on general-domain data would need revision.","The abstract's 'more than 14 BLEU' claim should be read against Section 5.1's computed average of 13.7 over the Vicuna-7B baseline; the aggregation that produces the larger figure is not shown.","Since the reward depends on off-the-shelf English-language scene-graph parsers and similarity models, the method's benefit may not transfer evenly to non-English source sentences; a natural experiment is to measure reward quality and translation gain per language."],"forward_implications":["Because the image is generated rather than retrieved or annotated, the method can be applied to any text-only MT corpus; no paired image data is needed at training or inference time.","The reported correlation between higher scene-graph reward and higher BLEU (Figure 4) implies that improving image-text consistency is itself a usable training signal for translation quality.","The paper's results on WMT24, especially En→Cs and En→Hi, imply that generated visual information can act as supplementary signal in low-resource translation directions where text-only LLMs struggle.","The method's cost is computational: the paper states that it requires collaborative training of the LLM and the diffusion model, so the translation gains come with a larger training budget."],"supporting_citations":[{"why":"Supplies the Stable Diffusion text-to-image model that IMAGE fine-tunes and uses to generate the source-sentence image.","marker":"Rombach et al., 2022"},{"why":"Provides the DDPO reinforcement-learning method used to fine-tune the diffusion model with the scene-graph reward.","marker":"Black et al., 2024"},{"why":"Defines the Multi30K benchmark and the paired image-text data that the method avoids needing.","marker":"Elliott et al., 2016"},{"why":"Supplies DreamLLM, the multimodal LLM base whose architecture IMAGE reuses and fine-tunes.","marker":"Dong et al., 2024"},{"why":"Provides Vicuna-7B, the underlying LLM for DreamLLM and the main text-only baseline.","marker":"Chiang et al., 2023"},{"why":"Supplies the CLIP visual encoder for image embeddings and the CLIP Score used to evaluate generated images.","marker":"Radford et al., 2021"},{"why":"Motivates the heuristic human-feedback reward as reinforcement learning from human preferences.","marker":"Ouyang et al., 2022"},{"why":"Supplies the CLIP Score metric used to measure similarity between generated images and source sentences.","marker":"Hessel et al., 2021"}],"fun_headline_variants":["Diffusion imagination lifts machine translation by 14 BLEU","No image annotations: diffusion-imagined scenes boost MT","Scene-graph reward aligns diffusion images with source text","Imagining sentences as images improves MT by 14 BLEU","Stable diffusion imagination aids multimodal and text-only translation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the WMT24 gains come from the generated visual information, but the paper provides no text-only baseline fine-tuned on the same Multi30K training data, so the visual signal's contribution is confounded with the benefit of additional translation fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion imagination lifts machine translation by 14 BLEU","No image annotations: diffusion-imagined scenes boost MT","Scene-graph reward aligns diffusion images with source text","Imagining sentences as images improves MT by 14 BLEU","Stable diffusion imagination aids multimodal and text-only translation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1617,"prompt_tokens":916,"completion_tokens":701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":618}},"tokens_in":532,"tokens_out":701,"duration_ms":6736,"temperature":1.0,"reasoning_tokens":618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:53:53.715390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a text-only Vicuna-7B model on the same Multi30K training pairs with the same $L_{\\mathrm{MLLM}}$ loss and training schedule, then evaluate it on the WMT24 test sets; if its BLEU/COMET scores are close to IMAGE's, the reported general-domain gains are not caused by the generated images.","supporting_citations":[],"review_version":1}