{"id":"319b0af0-cbff-47d4-8a5d-9b580abdae5b","arxiv_id":"2504.12140","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Fine-tuning translation LLMs on a curated multi-domain document dataset (DocBlocks) with document, chunk, and sentence level instruction formats improves document-level translation quality and speed while preserving sentence-level performance.","lead":"The authors fine-tune small open machine-translation models on DocBlocks, a new curated dataset of full documents, so the models can translate whole documents instead of one sentence at a time. The tuned small models match or beat much larger general chat models on long-document translation benchmarks across several language pairs, and translate faster than agent-based pipelines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never verifies that the IWSLT2017 and GuoFeng test documents are absent from DocBlocks, so the headline gains could be inflated by training/evaluation overlap.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern I would raise: training and evaluation draw from the same corpus families, and the paper never states that test documents were excluded from DocBlocks. This is not a stylistic or consensus disagreement; it is a concrete correctness risk that can be checked because DocBlocks is released. If the overlap check finds contamination, the headline comparison against 70B-scale baselines and agent-based methods is not trustworthy as evidence of document-level capability. If the check finds no overlap, the concern is retired. I agree with the CONDITIONAL verdict because the condition is explicit and verifiable, and the paper otherwise provides useful evidence: three base models, ablations, and a released dataset. Secondary weaknesses, such as the absence of human evaluation and significance testing, reinforce caution but do not change the verdict; the paper itself acknowledges that the automatic document-level metrics do not fully capture discourse quality. I therefore see no reason to move the reader's verdict.","tokens_in":29249,"tokens_out":3976,"duration_ms":42980,"concrete_test":"Download the released DocBlocks dataset and compute exact string matches and MinHash near-duplicates between every DocBlocks document and every document in the official IWSLT2017 and GuoFeng test sets, on both source and target sides; report overlap counts and overlap-level BLEU. If any overlap is found, retrain the three DocMT models on DocBlocks with all overlapping documents removed and recompute the GuoFeng and IWSLT2017 rows of Tables 2 and 3. If the gains persist after decontamination, the concern is resolved; if they shrink substantially, the central claim is largely a training-leakage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on a load-bearing premise the paper does not state: that the IWSLT2017 and GuoFeng test documents are excluded from DocBlocks. DocBlocks is explicitly built from IWSLT2016/2017 and GuoFeng (Section 2.1, Figure 2: 29.7K and 18.2K filtered documents), and Section 3.1 evaluates on the IWSLT2017 and GuoFeng test sets. The curation pipeline describes Bicleaner and CometKiwi filtering, langid checks, length and ratio thresholds, and deduplication, but it never describes removing benchmark test documents or near-duplicates of them. If overlap exists, the large gains in Tables 2 and 3 (e.g., GuoFeng d-BLEU 37.57 vs. 20.99 for Qwen2.5-72B) would partly reflect memorization of the evaluation corpus rather than a genuine document-level capability. This is a correctness and validity risk that a simple decontamination check would resolve.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DOCBLOCKS, a curated document-level parallel corpus, and a fine-tuning recipe that adapts sentence-level LLM translators (TOWER, EuroLLM, Qwen2.5) for document-level translation. The trained DocMT-LLMs are evaluated on GuoFeng and IWSLT2017 with document-level BLEU/COMET and LTCR, and on sentence-level benchmarks to check for catastrophic forgetting. The paper reports that the compact DocMT-LLMs surpass much larger prompting baselines on GuoFeng, are competitive on IWSLT2017, support chunked decoding with or without context and quality-aware reranking, and preserve sentence-level COMET within about 0.5 points.","tokens_in":29410,"tokens_out":3006,"duration_ms":30699,"significance":"If the central claims hold, the paper makes a useful contribution: it releases a document-level training corpus, shows that multi-granular instruction formats transfer across three base LLMs, and systematically contrasts fine-tuning with prompting and agent-based decoding. The ablations in Table 6 and the sentence-level stability results in Table 5 are valuable and clearly presented. The central empirical result, however, depends on the absence of training/evaluation overlap between DOCBLOCKS and the IWSLT2017 and GuoFeng test sets, which the paper does not currently establish; until that is resolved, the headline gains over much larger models cannot be taken at face value.","major_comments":[{"comment":"The discourse-quality claim ('coherent, high-quality document-level translations') rests entirely on automatic metrics (d-BLEU, d-COMET via SLIDE, LTCR), and the paper itself notes that no universally accepted document-level metric exists. Given that this is the paper's central contribution, the absence of any human evaluation of discourse coherence, terminology consistency, or anaphora makes the qualitative claim underdetermined. A small targeted human evaluation on GuoFeng and a subset of IWSLT2017 would substantially strengthen the conclusion.","section":"§3.1, §3.2"}],"minor_comments":[{"comment":"The sentence 'Sentence alignments are first computed using bleualign' does not cite the tool; please add the appropriate reference and a brief description of the alignment procedure.","section":"§3.1 Evaluation"}],"recommendation":"major_revision","confidential_remarks":"The training/test overlap issue is the decisive one. If the authors can convincingly show that DOCBLOCKS excludes the IWSLT2017 and GuoFeng test documents, the paper's core message and ablations are strong enough for publication at a venue like COLM. The speed claim versus agents should be corrected or measured, and a human-evaluation add-on would materially improve the discourse-quality claim. The paper is otherwise well within scope for a machine translation venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful systems paper with a real but fixable gap. The new things—DocBlocks, the multi-paradigm SFT recipe, the three-backbone comparison, and the ablations—are worth having. The gap is that the curation section never says the IWSLT2017 and GuoFeng test documents were held out of DocBlocks, even though both corpora are built from the same source families. That is the first question a referee should ask.\n\nWhat the paper does well: the evaluation breadth is genuine. Three base models (Tower, EuroLLM, Qwen2.5-7B) are tested with sentence-level retention checks, throughput numbers, and ablations that isolate filtering, MRD2D, CAPT, and the sentence-data share. The results are internally consistent: fine-tuning on DocBlocks lifts document-level metrics relative to prompting the same base models, and the gains hold across Doc2Doc and chunked decoding. The claim that 7B-9B models beat 70B prompting baselines on GuoFeng is credible if the test data is clean.\n\nThe soft spots are mostly proportional. The decontamination ambiguity is the biggest one. DocBlocks is built from IWSLT2016/2017 and GuoFeng, and evaluation uses the IWSLT2017 and GuoFeng test sets, but the paper never states that test documents or near-duplicates were removed. This is not an exotic worry: TED transcripts and web novels can cross-corpus share, and some gains (GuoFeng d-BLEU from 20.99 to 37.57 for Qwen2.5-72B) are large enough that part could come from training/evaluation overlap. A simple n-gram overlap check or a held-out subset would resolve it. Second, all metrics are automatic; d-COMET and LTCR are weak proxies for discourse quality, and the paper itself concedes this. Third, there is no significance testing or variance reporting, so gaps of a point or two on d-COMET may be noise. The agent comparison is also partially confounded: TRANSAGENTS and DELTA use different base models (GPT-4 Turbo, GPT-4o mini), so the measured difference mixes method with model capability.\n\nOverall, the central claim is plausible and the dataset is a useful resource. I would send this to peer review; the main revision request would be a decontamination check plus reporting of variance across runs. The paper is for MT researchers who want a modern doc-level fine-tuning recipe and for anyone building document-level datasets.","headline":"A useful, honestly reported doc-level SFT recipe with a real but fixable gap: the paper never states that the IWSLT2017 and GuoFeng test documents were excluded from DocBlocks, so the headline gains should be treated as provisional until a decontamination check is done.","tokens_in":30096,"tokens_out":2204,"would_cite":true,"duration_ms":23270,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on DocBlocks lets small open models beat 70B and GPT-4o at document-level translation.","keywords":["document-level machine translation","large language models","supervised fine-tuning","context-aware prompting","parallel corpus curation","discourse coherence","chunked decoding"],"falsifier":"Run exact and near-duplicate detection between the IWSLT2017 and GuoFeng test documents and the released DocBlocks training corpus; if any test document, chunk, or near-duplicate appears in training, or if retraining after removing these near-duplicates erases the reported d-BLEU gains, the claim of document-level generalization is not supported.","tokens_in":28948,"feed_emoji":"🌐","tokens_out":8078,"duration_ms":76892,"temperature":0.7,"pith_summary":"The paper tries to establish that document-level machine translation does not require larger models or multi-agent pipelines: targeted fine-tuning of existing instruction-tuned LLM translators on a curated corpus of full documents and contextual chunks is enough. It introduces DocBlocks, a cleaned multilingual parallel corpus drawn from news, TED talks, parliamentary proceedings, and novels, and trains three 7B-9B open models on a mix of document-to-document, context-windowed chunk, and sentence-level instructions. If the paper is right, small open models can produce coherent whole-document translations that beat 70B-parameter prompting baselines and GPT-4o on a discourse-heavy benchmark, while retaining sentence-level translation quality.","feed_headline":"Fine-tuning lifts 7B translators past 70B rivals","feed_subtitle":"One curated multilingual corpus teaches small open models document-level coherence without hurting sentence quality.","key_machinery":"The central object is the DocBlocks training corpus and its instruction formats. DocBlocks pairs full documents from five domains (News Commentary, IWSLT, Europarl, GuoFeng, BWB) with document-level alignment, quality filtering, deduplication, and length-based cleaning. Two training techniques carry the argument: MRD2D splits each document into k parts with k in {1, 2, 4} to create multi-resolution document-to-document examples, and CAPT adds a context window of up to three previously translated chunks to chunk-level prompts, alongside standalone sentence-level examples. The loss is computed only on target tokens, and the same trained model can be decoded whole-document or chunk-by-chunk.","core_discovery":"The paper's central claim is that a single supervised fine-tuning pass over multi-granular document-level data turns sentence-level LLM translators into document-level translators. The resulting DocMT-LLMs outperform their untuned counterparts and, despite being 7B-9B models, beat Llama-3.3-70B-Instruct, Qwen2.5-72B-Instruct, and GPT-4o on GuoFeng zh→en and remain competitive on IWSLT2017, while keeping sentence-level COMET within about 0.5 points of the base models. The authors attribute this to training on a mixture of document-to-document, context-windowed chunk, and sentence-level formats, which teaches the model to use surrounding context during decoding without sacrificing single-sentence ability.","pith_inferences":["Beyond the paper: if DocBlocks is later shown to contain the evaluation documents or near-duplicates, the reported gains would mostly reflect domain adaptation rather than generic document-level capability; testing on held-out domains such as legal or medical documents would settle this.","Beyond the paper: the same multi-granular recipe of full documents, k-way chunks, and windowed context may transfer to other instruction-tuned multilingual LLMs and to other discourse-sensitive tasks such as long-document summarization.","Beyond the paper: the comparison with agent-based methods suggests that hierarchical memory agents buy little once the base model is document-trained; a direct study of where agent memory helps, such as very long novels with recurring characters, would be a natural next test."],"forward_implications":["A 7B or 9B open model can serve as a document-level translation engine that outperforms 70B-class prompting models on discourse-rich literary translation and matches them on TED-talk documents.","The same model can decode whole documents in one pass or in parallel chunks, so users can trade quality for throughput; standard chunking reaches 392.45 tokens/s versus 204.22 for document-to-document.","Document-level training on a curated corpus does not erase sentence-level translation ability; COMET drops are around 0.5 points, and Qwen2.5-7B improves at sentence level after training on DocBlocks.","Agent-based translation pipelines and prompt-only context engineering are not necessary for coherent document translation; direct fine-tuning plus simple context windows matches or beats them."],"supporting_citations":[{"why":"Supplies the IWSLT TED-talk corpus used both as a DocBlocks source and as the IWSLT2017 evaluation benchmark.","marker":"Cettolo et al., 2017"},{"why":"GuoFeng provides the discourse-rich novel corpus used as a DocBlocks source and as the zh→en test set.","marker":"Wang et al., 2023b"},{"why":"Europarl supplies parliamentary proceedings data included in DocBlocks.","marker":"Koehn, 2005"},{"why":"BWB supplies the web-novel parallel data included in DocBlocks.","marker":"Jiang et al., 2022"},{"why":"Provides the multi-resolution document-to-document idea of splitting documents into k parts used as MRD2D training.","marker":"Sun et al., 2022"},{"why":"Defines the context-aware prompt tuning format with preceding source-target chunks that DocBlocks adapts as CAPT.","marker":"Wang et al., 2023a"},{"why":"TowerInstruct-Mistral-7B is one of the three base models fine-tuned into DocMT-LLMs.","marker":"Alves et al., 2024"},{"why":"EuroLLM-9B-Instruct is a second base model used to test the method's generalizability.","marker":"Martins et al., 2024"},{"why":"Qwen2.5 models supply both a 7B base model for fine-tuning and the 72B prompting baseline.","marker":"Bai et al., 2023"},{"why":"SLIDE provides the sliding-window method used to compute document-level COMET for the main results.","marker":"Raunak et al., 2024"}],"fun_headline_variants":[],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the IWSLT2017 and GuoFeng test documents are not present in DocBlocks, since the corpus is built from those same collections and the paper never states that the evaluation documents were excluded.","fun_headline_variants_meta":{"error":"Client error '402 Payment Required' for url 'https://api.deepseek.com/chat/completions'\nFor more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402"},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:36:42.289292+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run exact and near-duplicate detection between the IWSLT2017 and GuoFeng test documents and the released DocBlocks training corpus; if any test document, chunk, or near-duplicate appears in training, or if retraining after removing these near-duplicates erases the reported d-BLEU gains, the claim of document-level generalization is not supported.","supporting_citations":[{"cited_title":"E uroparl: A parallel corpus for statistical machine translation","cited_arxiv_id":null,"evidence_quote":"Europarl supplies parliamentary proceedings data included in DocBlocks."}],"review_version":1}