{"id":"f9055041-7510-4df4-922a-f02c10e741ef","arxiv_id":"2507.08309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-tuning paradigm that prompts MLLMs to self-generate OCR text before translating document images improves DIMT quality and reduces catastrophic forgetting of OCR.","lead":"This paper introduces a fine-tuning method, SSR, that makes multimodal LLMs generate an OCR transcript of a document image before translating it, instead of translating the image directly. On document image machine translation benchmarks, SSR improves translation quality and cross-domain generalization while largely preserving the model's original OCR ability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Frozen self-generated OCR transcript is the load-bearing bridge, but the paper never measures whether the fine-tuned model's online OCR prefix still matches it.","rationale":"Good-faith reading: the paper's empirical contribution is a training scheme that concatenates a model's own OCR output with the target translation and fine-tunes an MLLM on that sequence; the evidence in Tables 1, 2, 4, 5, and 7 is broadly consistent across four MLLMs, two cross-domain test sets, and three language pairs, which makes the core empirical claim credible. The most load-bearing assumption is not whether the method beats SFT(DIMT) on these specific benchmarks, but whether the frozen X' remains the right conditioning signal after fine-tuning. The paper states this as a design principle (Section 3.1: 'sampled from the MLLM's original distribution') but never verifies that the fine-tuned model, at inference, actually produces the same OCR prefix under the combined prompt. This is exactly the reader's weakest assumption. I agree with the reader that this makes acceptance conditional. I do not think it warrants rejection: the OCR-retention numbers in Table 2 suggest the model's OCR behavior is largely preserved, and the method's gains are consistent and often large. But because the core mechanism is asserted rather than measured, the honest verdict remains CONDITIONAL, not ACCEPT. Error bars and multiple seeds would help, but the missing online-vs-frozen OCR comparison is the specific test that would settle the central mechanism. No independent formal verification exists; that is consistent with the field norm and does not by itself change the verdict.","tokens_in":22877,"tokens_out":7052,"duration_ms":82470,"concrete_test":"Run the published SSR pipeline on Qwen2-VL (or Vary-base) with the released code. On a held-out set of 200 DoTA images, sample X' from the base model under P_ocr exactly as in Eq. 1. Then, after the 3-epoch fine-tuning, decode the full SSR response under P_combined, extract the OCR prefix X'_final, and also sample X'_0 from the base model under P_combined. Report (a) character accuracy and edit distance between X' and X'_final, and between X' and X'_0; (b) translation BLEU under three conditioning regimes: teacher-forced prefix X', free-decoded prefix X'_final, and base-model combined-prompt prefix X'_0. If CA(X', X'_final) is high and BLEU(X'_final) is not below BLEU(X'), the concern is resolved. If BLEU(X'_final) drops materially or the prefixes diverge, the frozen-transcript assumption fails and the central mechanism needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism (Eqs. 1-3) freezes X' ~ MLLM(P_ocr, I) before training and uses R = CONCAT(X', <Translation>, Y) as the supervised target for all three epochs. At inference, the model is prompted with P_combined = 'Convert the content in the image to Markdown, then translate into Chinese' and must autoregressively produce its own OCR prefix before translating. The training-time distribution over X' is therefore p_{theta0}(X' | P_ocr, I), while the inference-time distribution is p_{theta_final}(X' | P_combined, I). Two unmeasured mismatches follow: (1) P_combined differs from P_ocr even at epoch 0, so the sampled X' may not be on-distribution for the actual training prompt; (2) LoRA fine-tuning shifts theta, so the online OCR prefix can drift away from the frozen X' used to train the translation head. Table 2 evaluates the final model's OCR only under the original OCR prompt, not under P_combined, and no experiment compares the final model's emitted OCR prefix with the frozen X'. This matters because if the online prefix drifts, the translation head is conditioned on a stale source text; the reported BLEU gains could then come from the model learning to bypass X' or from re-learned OCR, rather than from the 'smooth convergence' mechanism claimed in Section 3. Table 4 varies the source-text type but does not test the actual online-vs-frozen match, so it does not resolve this. The claim is still plausible, but this bridging assumption is the least secured load-bearing step.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SSR, a fine-tuning strategy for document image machine translation (DIMT) with multimodal large language models. For each training image, the base MLLM is first prompted with its original OCR instruction to sample a source-text transcript X' (Eq. 1); the training target concatenates X', a special token, and the ground-truth translation (Eq. 3), and the model is fine-tuned with the negative log-likelihood over this response (Eq. 2). At inference the model is prompted to 'convert the image to Markdown, then translate into Chinese', so it generates its own OCR prefix before translating. Experiments with Vary-toy/Vary-base, Textmonkey, and Qwen2-VL, fine-tuned with LoRA on 10K DoTA samples, report that SSR improves DIMT BLEU/BLEU-PT/STEDS relative to SFT and other baselines on in-domain DoTA and zero-shot DITrans test sets, while largely preserving OCR and VQA performance; extensions cover unsupervised data, monolingual-task selection, low-resource training, and other language pairs.","tokens_in":23230,"tokens_out":6136,"duration_ms":68431,"significance":"If the reported effects are robust, SSR is a practical and inexpensive way to mitigate catastrophic forgetting in DIMT fine-tuning: it requires no auxiliary data beyond the model's own OCR samples, works across four architectures of different sizes, and the gains are consistent and often large. The paper also provides useful ablations (source-text type, monolingual task, data scale) and makes code available. However, the central mechanism rests on an unverified assumption that the frozen pre-training OCR transcript remains representative of the OCR prefix the fine-tuned model generates at inference; the manuscript does not measure this drift, so the key causal story is not yet fully supported.","major_comments":[{"comment":"The load-bearing assumption of the method is that the frozen self-generated transcript X' used during training remains representative of the OCR prefix that the fine-tuned model produces at inference. Equation (1) samples X' from the base model under P_ocr before fine-tuning, while at inference the model is prompted with P_combined ('... then translate into Chinese') and must autoregressively emit its own OCR prefix from the updated parameters. The paper never measures the agreement between the online OCR prefix and the frozen X'. Table 2 evaluates OCR only under the original OCR prompt, not under P_combined; Table 4 varies the source-text type but does not test the online-vs-frozen match. Consequently, the 'smooth convergence' explanation in Section 5.1 and Figure 4 is not the only possible account of the BLEU gains; the model could partially bypass X' or re-learn OCR, and the reported gains would then not validate the stated mechanism. Please add a direct measurement (e.g., edit distance or token-level agreement between the fine-tuned model's OCR prefix under P_combined and the frozen X' used for the same images) and, ideally, an ablation that conditions inference on the frozen X' versus the model-generated prefix. The Limitations section should also acknowledge this assumption.","section":"Sec. 3.1-3.2, Eqs. (1)-(3); Sec. 5.1; Fig. 4"},{"comment":"Cross-domain test sets are small (100 images per domain after merging Ads & News), and no error bars, multiple seeds, or significance tests are reported. Given that a central claim is large zero-shot cross-domain improvements (e.g., Qwen2-VL Ads & News +10.13 BLEU over SFT(DIMT) in Table 1), the absence of variance estimates is material: BLEU computed on 100 documents has substantial sampling noise. Please report confidence intervals, significance tests, or results over multiple seeds, or enlarge the cross-domain evaluation sets.","section":"Sec. 4.1, App. A.1; Tables 1 and 5"}],"minor_comments":[{"comment":"The header contains a typo: 'BELU' should be 'BLEU'.","section":"Table 8"},{"comment":"In the CoT (Cascade) prompt template, the line '(original image caption instruction of the MLLM)' appears to be a copy-paste error; the instruction shown is the OCR instruction, not an image caption instruction.","section":"App. A.3"},{"comment":"The code repository URL is garbled by escape sequences in the rendered text; please provide a clean, clickable URL.","section":"Footnote 1"},{"comment":"The method is called 'Synchronously Self-Reviewing' in the title and abstract but 'Synchronized Self-Reviewing' in the introduction; please use one name consistently.","section":"Abstract and Sec. 1"},{"comment":"The claim that Vary-base achieves 'performance parity' with ground-truth versus self-generated source text is not fully supported by the table: in the Political Report (CD) column, SSR w Ground Truth Text gives 26.05 BLEU while SSR w Self-generated Text gives 21.47 BLEU, a 4.58-point difference; this discrepancy should be acknowledged or discussed.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CL and the method is simple enough to reproduce. My main reservation is the unverified online-prefix versus frozen-transcript match; if the authors add the suggested measurements, I would be willing to accept after minor revisions. The small cross-domain test sets without significance testing should also be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the core idea is simple and mostly new: sample the base MLLM's self-generated OCR transcript for each image, concatenate it with a <Translation> token and the reference, and fine-tune on that sequence. It is replay-based continual learning, but the replay data is the model's own OCR output, which makes it cheap and model-specific. Second, the experiments are unusually thorough: four MLLMs, three language pairs, cross-domain tests, commercial API comparisons, ablations on source-text type, monolingual task choice, low-resource settings, and unsupervised data. The gains over SFT are consistent and often large, especially cross-domain, and the OCR preservation is striking (Qwen2-VL drops only 0.12 CA on document OCR versus collapse to 5.96 for SFT). For the practical question, does this recipe work, the answer is yes.\n\nThe main soft spot is real but specific: the method freezes the OCR transcript sampled before training and never checks whether the fine-tuned model's online OCR prefix at inference matches that frozen distribution. The prompt used at inference differs from the training prompt, and LoRA fine-tuning could shift OCR behavior. The 'smooth convergence' claim in Section 3 is plausible but not directly evidenced. Table 4 varies the source text type at training time, not the online-vs-frozen match.\n\nTwo smaller issues: no error bars or multiple seeds, and the cross-domain test sets are only 100 images each, so single-point BLEU differences of a few points are hard to interpret. The SFT(DIMT) baseline is also an easy target for OCR forgetting, since it trains only on translations.\n\nNone of this sinks the paper. The recipe works, the ablations are sensible, and the unsupervised-data extension is a nice touch. I would send this to reviewers, it deserves a serious referee, but I would ask for a direct drift analysis and at least one seeded run. For reading group, it is a good example of self-replay on multimodal models, with the missing drift check as the discussion point.","headline":"A cheap, well-tested fine-tuning recipe for DIMT that preserves OCR, but the load-bearing assumption that the frozen self-generated OCR transcript stays on-distribution after training is never directly checked.","tokens_in":800,"tokens_out":1282,"would_cite":true,"duration_ms":21689,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on the model's own OCR transcript improves document image translation and preserves OCR ability.","keywords":["document image machine translation","multimodal large language models","optical character recognition","catastrophic forgetting","supervised fine-tuning","self-review","cross-lingual generalization","low-resource translation"],"falsifier":"During SSR fine-tuning, take the training images and regenerate the OCR transcript with the partially fine-tuned model at each epoch; if the regenerated transcript drifts from the frozen $X'$ (e.g., character accuracy of regenerated versus frozen text falls) and translation BLEU on a held-out set simultaneously degrades, the stale-transcript assumption is falsified. A direct test: an online variant that resamples $X'$ each epoch from the current model should beat the frozen version if drift matters.","tokens_in":22698,"feed_emoji":"📄","tokens_out":4884,"duration_ms":43505,"temperature":0.7,"pith_summary":"MLLMs are good at OCR but bad at translating the text inside document images, and standard supervised fine-tuning on translation data makes them forget how to read. This paper proposes SSR: before fine-tuning, prompt the model to produce its own OCR transcript of each training image, then train it on that transcript followed by the ground-truth translation. The authors claim this joint objective improves translation quality over standard SFT on all four MLLMs tested, sharply reduces catastrophic forgetting of OCR and VQA abilities, and improves zero-shot translation on unseen document domains. The practical payoff is that document image translation can be improved without sacrificing the model's monolingual competence, using only the model's own OCR output plus target translations.","feed_headline":"Own OCR output improves document translation and stops forgetting","feed_subtitle":"Fine-tuning on OCR transcript plus target translation beats plain fine-tuning and keeps OCR ability intact.","key_machinery":"The central object is the SSR-constrained prompt template, which appends the original OCR instruction (e.g., \"Convert the content in the image to Markdown\") with \"then translate into Chinese,\" and sets the expected response to be the frozen self-generated OCR transcript $X'$, the special token $\\langle\\text{Translation}\\rangle$, and the ground-truth target $Y$. Training minimizes the negative log-likelihood of this response given the image and instruction. Because $X'$ is sampled from the base model's own output distribution, the training signal stays close to the model's existing monolingual behavior while teaching the cross-lingual image-to-translation mapping; the authors argue this yields a smoother loss curve and preserves OCR proficiency.","core_discovery":"On its own terms, the paper's central discovery is that the target for DIMT fine-tuning should not be the translation alone but the concatenated sequence $R = \\mathrm{CONCAT}(X', \\langle\\text{Translation}\\rangle, Y)$, where $X'$ is the OCR transcript the base model itself generates from the document image and $Y$ is the ground-truth translation. Training with the standard negative log-likelihood loss on this sequence lets the model condition translation on its own OCR output. Evidence includes Qwen2-VL reaching 57.23 in-domain BLEU versus 53.92 for SFT, and retaining 85.18 document OCR character accuracy versus 5.96 for SFT; similar trends hold for Vary-base, Vary-toy, and Textmonkey. The paper further shows the method transfers cross-domain, extends to other language pairs, and can use unsupervised document images plus machine-translated synthetic targets to gain additional improvements.","pith_inferences":["The method is effectively a self-distillation of the OCR stream: it anchors the fine-tuned model to its own pre-fine-tuning image-to-text behavior, so what is preserved is not the OCR training set but the model's own output distribution.","The same 'strong retained skill as prefix' recipe could be applied to other multimodal fine-tuning targets, e.g., inserting a captioning or grounding prefix before a new task output to protect those abilities.","A testable extension the paper does not run: resample the OCR transcript online during training from the evolving model; if that improves translation further, the fixed-transcript assumption is the active bottleneck.","The analogy to bilingual cognitive advantage suggests a general fine-tuning principle: chain a familiar monolingual, monotask generation step before a new cross-lingual, cross-modal step to keep both skills."],"forward_implications":["Standard SFT for DIMT can be replaced by SSR to obtain higher BLEU, BLEU-PT, and STEDS on in-domain and zero-shot cross-domain tests across four MLLMs.","Catastrophic forgetting of OCR is largely avoided: for example, Qwen2-VL retains 85.18 document OCR character accuracy versus 5.96 under SFT, and SSR even improves scene-text OCR over the base model.","The method transfers to other language pairs (English-French, English-German) and is effective in low-resource settings, with SSR at 10K training samples beating SFT trained on 100K samples by 3.01 in-domain BLEU.","Unsupervised document images can be turned into training data by OCR-ing them with the MLLM and machine-translating the transcript, further improving in-domain and cross-domain translation.","Fine-tuning with SSR also induces cross-lingual VQA generalization: the model answers Chinese questions on English document images zero-shot."],"supporting_citations":[{"why":"Provides the DoTA dataset used for training and in-domain evaluation, the DIMTDA baseline, and the metric computation procedure.","marker":"Liang et al., 2024"},{"why":"Provides the DITrans dataset used for zero-shot cross-domain tests.","marker":"Zhang et al., 2023b"},{"why":"Defines the standard supervised fine-tuning paradigm that SSR is compared against (SFT baselines).","marker":"Ouyang et al., 2022"},{"why":"Defines the chain-of-thought prompting baselines CoT (Direct) and CoT (Cascade).","marker":"Wei et al., 2022"},{"why":"Provides the SDFT self-distillation/replay baseline that also generates self-training data.","marker":"Yang et al., 2024b"},{"why":"Provides Qwen2-VL, one of the four MLLMs, and the original OCR prompt template used to construct SSR.","marker":"Wang et al., 2024"},{"why":"Provides Vary-base, the MLLM used for the loss-curve analysis and low-resource experiments.","marker":"Wei et al., 2024a"},{"why":"Provides Textmonkey, one of the four MLLMs evaluated.","marker":"Liu et al., 2024"}],"fun_headline_variants":["Self-generated OCR transcript boosts document translation and curbs forgetting","Train on OCR plus translation to improve document translation and retain OCR","Concatenate OCR and translation for fine-tuning to stop forgetting in MLLMs","OCR self-review improves document translation and keeps OCR ability intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method freezes the self-generated OCR transcript $X'$ before fine-tuning and assumes this transcript still matches the distribution the model will produce after fine-tuning, so the learned image-to-transcript-to-translation mapping is trained on a representative source text.","fun_headline_variants_meta":{"raw":{"variants":["Self-generated OCR transcript boosts document translation and curbs forgetting","Train on OCR plus translation to improve document translation and retain OCR","Concatenate OCR and translation for fine-tuning to stop forgetting in MLLMs","OCR self-review improves document translation and keeps OCR ability intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000617,"raw_usage":{"total_tokens":2848,"prompt_tokens":911,"completion_tokens":1937,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":1864}},"tokens_in":527,"tokens_out":1937,"duration_ms":13930,"temperature":1.0,"reasoning_tokens":1864,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:23:20.218629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"During SSR fine-tuning, take the training images and regenerate the OCR transcript with the partially fine-tuned model at each epoch; if the regenerated transcript drifts from the frozen $X'$ (e.g., character accuracy of regenerated versus frozen text falls) and translation BLEU on a held-out set simultaneously degrades, the stale-transcript assumption is falsified. A direct test: an online variant that resamples $X'$ each epoch from the current model should beat the frozen version if drift matters.","supporting_citations":[],"review_version":1}