{"id":"8ffb3b8a-07c8-4d05-b043-2f9c0fb91497","arxiv_id":"2607.24515","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Dataset quality and alignment dominate English–Tamil NLLB/mBART scores; attention entropy tracks quality, and few-shot TamilLaMA is fluent but less literal than supervised NMT.","lead":"This paper benchmarks NLLB and mBART on English–Tamil translation across four datasets of different quality, adds attention heatmaps, and qualitatively compares few-shot TamilLaMA. It mainly confirms that clean parallel data drives scores and that noisy mined corpora can inflate BLEU.","discovery_kind":"incremental","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The PMIndia near-zero result — the paper's strongest evidence that dataset quality dominates — is anomalous (chrF 0.51 implies essentially zero character overlap) and is more consistent with an evaluation/preprocessing failure than with corpus noise.","rationale":"The reader flagged the inference-only attribution gap (dataset noise vs. pretraining-domain mismatch, plus unstated preprocessing details). My concern is adjacent but sharper and more concrete: not merely that the attribution is under-controlled, but that the single most dramatic data point supporting the attribution — PMIndia chrF ≈ 0.5 — is quantitatively implausible as a data-noise effect and carries internal fingerprints of an encoding/alignment failure in the evaluation pipeline (corrupted Tamil glyphs in the paper's own examples; the paper's own PMIndia PB-SMT baseline at BLEU 9.56; a 3× NLLB-vs-mBART BLEU discrepancy on WikiMatrix that goes unexplained). I agree with the reader that the inference-only design confounds noise with domain fit, so agreement is \"partial\": same region of the argument, different specific failure point. The reader's CONDITIONAL verdict is the right tier; I keep it CONDITIONAL rather than REJECT because the artifact hypothesis is falsifiable with the cheap check above and the qualitative direction of the finding (clean aligned data helps) is consistent with prior work. But the conditions should explicitly include the PMIndia audit and an explanation of the WikiMatrix BLEU/chrF inversion, since the conclusion's most quotable sentence (\"near-zero scores on a noisy dataset like PMIndia\") currently rests on numbers that look broken rather than informative.","tokens_in":9505,"tokens_out":2317,"duration_ms":82184,"concrete_test":"Re-run NLLB EN→TA and TA→EN on a random 200-sentence PMIndia subset: verify sentence-ID alignment between hypotheses and references, apply NFC Unicode normalization to both, score with standard sacreBLEU/chrF, and manually inspect 10 outputs for script corruption or empty strings. If chrF rises from ~0.5 to >15, the near-zero result is an evaluation artifact and the headline claim must be reframed; if chrF stays ~0.5 with visibly correct outputs and references, the noise attribution stands.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (\"quality and alignment of the datasets play a major role... near-zero scores when using a noisy dataset like PMIndia,\" Conclusion; Table 2) leans hardest on the PMIndia row: NLLB and mBART allegedly score BLEU 0.25/0.35 and chrF 0.44–1.14 in both directions. A chrF of ~0.5 is not \"noisy data\" territory — even a badly domain-mismatched model producing grammatical text in the right language and script will almost always exceed chrF 15–25. chrF ≈ 0.5 means near-zero character n-gram overlap, which happens when (a) hypotheses and references are misaligned (wrong sentence IDs/order), (b) script/encoding corruption makes the two strings disjoint, or (c) outputs are empty/in the wrong language. Three internal signals point to a pipeline artifact rather than corpus noise: (1) the paper's own Table 2 reports PB-SMT on PMIndia achieving BLEU 9.56 (Thapliyal et al., 2022), so the corpus is demonstrably usable; PMIndia is human-translated government text — domain-mismatched, yes, but not \"very high noise\" in the WikiMatrix mined sense; (2) the Tamil strings quoted in §4.2.2 and §4.3.2 contain visibly corrupted/mixed-script glyphs (e.g., \"அரா ழராமப்௖ற பஶொக௬க்கான\"), indicating Unicode handling problems somewhere in preprocessing or rendering — if that corruption touched either hypotheses or references, BLEU/chrF would collapse exactly as observed; (3) the WikiMatrix row shows the opposite anomaly (mBART EN→TA BLEU 28.21 vs NLLB 9.88 on identical data, and BLEU far above the \"clean\" EnTamV2 scores), which the paper waves away as \"inflated BLEU\" — together suggesting the metric/eval pipeline, not just the data, is unstable. If the PMIndia near-zero is an artifact, the paper's most dramatic support for \"dataset noise dominates\" evaporates, leaving only modest EnTamV2-vs-NTREX differences that confound data quality with model-domain fit (the reader's concern).","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The manuscript evaluates two off-the-shelf multilingual NMT models (NLLB, mBART) and a few-shot prompted Tamil-adapted LLaMA on English↔Tamil translation across four parallel corpora of differing quality (EnTamV2, NTREX, WikiMatrix, PMIndia), using BLEU and chrF, plus a cross-attention interpretability analysis with an attention-entropy measure. The central claim is that dataset quality and domain alignment dominate performance: well-aligned EnTamV2 yields consistent gains, while PMIndia allegedly yields near-zero scores; lower attention entropy is claimed to track translation quality; few-shot TamilLaMA is claimed to produce fluent conversational translations while lagging on literal accuracy. The directional finding (clean, well-aligned data beats noisy, mismatched data) is plausible and consistent with most of Table 2, but the paper's single strongest evidence row — PMIndia — exhibits chrF ≈ 0.5, which implies near-zero character overlap and is the signature of a preprocessing/alignment/encoding failure rather than corpus noise; the paper's own corrupted Tamil strings and its duplicated PB-SMT baseline number reinforce this concern. Several results (WikiMatrix mBART 28.21 BLEU vs NLLB 9.88 on identical data) are anomalous and unexplained because checkpoints, preprocessing, and decoding details are undisclosed.","tokens_in":9858,"tokens_out":3826,"duration_ms":134076,"significance":"Systematic, reproducible benchmarking for genuinely low-resource pairs like English–Tamil is valuable to the community, and the paper attempts something broader than a single-model report: per-dataset comparison, an attention-entropy diagnostic, and a few-shot LLM comparison. If the PMIndia anomaly is diagnosed and the checkpoints/preprocessing are disclosed, the finding that off-the-shelf multilingual NMT performance tracks corpus provenance and domain would be a useful, falsifiable empirical result. However, the current version's headline numbers do not yet support its conclusions: its strongest evidence row has the signature of a pipeline failure, and no error bars, significance tests, checkpoint details, or reproducible decoding configurations are provided. The contribution is presently closer to an exploratory study than a systematic analysis.","major_comments":[{"comment":"The PMIndia row is the load-bearing evidence for the central claim ('near-zero scores when using a noisy dataset like PMIndia', Conclusion), and it is far more consistent with an evaluation/preprocessing failure than with corpus noise. chrF of 0.44–1.14 implies essentially zero character n-gram overlap between hypothesis and reference; even a severely domain-mismatched model emitting grammatical text in the correct language and script virtually always exceeds chrF 15–25. Three internal signals point to an artifact: (i) the same table reports PB-SMT on PMIndia at BLEU 9.56 (Thapliyal et al., 2022), so the corpus is demonstrably usable; PMIndia is human-translated government text — domain-mismatched, yes, but not 'very high noise' in the mined-corpus sense; (ii) the Tamil strings quoted in §4.2.2 and §4.3.2 contain visibly corrupted/mixed-script glyphs (e.g., 'அரா ழராமப்௖ற பஶொக௬க்கான'), in","section":"Table 2, PMIndia row; §4.1.3; Conclusion"},{"comment":"The PB-SMT (Thapliyal et al., 2022) baseline is listed at exactly BLEU 9.56 for both the WikiMatrix row and the PMIndia row. This duplication is almost certainly a copy error — the two corpora are distinct — and one or both attributions are wrong. Separately, Table 2 mixes prior-work numbers (MIDAS, Transformer-NMT, PB-SMT) with the authors' own scores without stating whether tokenization, BLEU implementation, or test splits are comparable; Post (2018) is cited but no sacreBLEU signature is given. The baseline rows must be corrected and comparability conditions stated, since several discussion claims (§4.1.1, §4.1.3) lean on these comparisons.","section":"Table 2, WikiMatrix and PMIndia rows"},{"comment":"On WikiMatrix EN→TA, mBART is reported at BLEU 28.21 versus NLLB at 9.88 on identical data — a 3× gap between two off-the-shelf multilingual models — and TA→EN scores of 31–40 BLEU are far above published off-the-shelf numbers for this pair on noisy mined data. The paper never states which checkpoints were used (mBART-base vs mBART-50, and which fine-tuned direction), which language tags were forced, or the exact preprocessing. A gap this large strongly suggests a configuration difference (e.g., a fine-tuned mBART-50-many-to-one checkpoint versus a mismatched NLLB language tag), not a model-quality difference. Checkpoint IDs, decoding hyperparameters (beam size, length penalty values), and tokenization must be reported; otherwise neither the WikiMatrix anomaly nor the cross-model comparison is interpretable.","section":"Table 2, WikiMatrix row; §3.3"},{"comment":"The study runs NLLB and mBART strictly in inference mode ('without performing full fine-tuning but only decoding optimization', §3.3), yet attributes the cross-dataset score gaps primarily to dataset noise and alignment (§4.1.3, Table 3, Conclusion). With inference-only evaluation, score differences confound corpus noise with (a) the match between each corpus's domain and the models' pretraining/fine-tuning data (NLLB was trained partly on NLLB-Seed and WikiMatrix-adjacent mined data) and (b) unstated checkpoint differences. The noise-level labels in Table 3 are themselves qualitative and partly circular: WikiMatrix is labeled 'High noise / Low alignment' yet yields the highest BLEU/chrF scores in the whole study, while PMIndia — human-translated — is labeled 'Very High noise' on the basis of the anomalous near-zero row. At minimum the claims must be hedged to 'off-the-shelf performance","section":"§3.3; §4.1.3; Table 3"},{"comment":"§4.3.4 claims 'a consistent relation between the attention entropy and the translation performance' and the Conclusion elevates entropy to 'a quantitative way to measure the alignment of the dataset.' But (i) no entropy numbers are reported anywhere — Figure 6 is the only evidence and no correlation coefficient or per-dataset means are given; (ii) cross-attention entropy is a property of the model's decoding behavior, not of the dataset, so treating it as a dataset-alignment measure conflates model uncertainty with data noise; (iii) the literature (e.g., the attention-is-not-explanation line of work) cautions against reading alignment directly off attention weights, a caveat the paper does not acknowledge despite citing Ghader & Monz (2017). The section needs actual entropy values per dataset, a reported correlation with BLEU/chrF (with n and p), and reframed language; as written the int","section":"§4.3.3–4.3.4; Figure 6; Conclusion"},{"comment":"The TamilLaMA experiments are underspecified to the point of irreproducibility, and contain an internal contradiction: §4.2.1 says the model 'has been specifically fine-tuned/adapted' to Tamil, while §4.2.2 says 'No fine-tuning was done.' The checkpoint/adapter used, number of few-shot exemplars, exemplar selection procedure, prompt template, and generation settings are all unstated. Table 4 mixes previously reported IN22 numbers with the authors' own 'IN22 (evaluated)' rows without stating whether the evaluation split, reference set, and metric implementation match — so the claim that TamilLaMA is competitive with GPT-3.5/LLaMA-2-13B is not on a controlled footing. Additionally, the few-shot results are evaluated qualitatively only (Limitations acknowledges this), yet the abstract claims in-context prompting is 'an excellent way' to perform translation. The setup must be fully specified","section":"§4.2; Table 4"}],"minor_comments":[{"comment":"Figure 4 is captioned 'Attention HeatMap of Tamil to English Translation' but the token mapping given underneath lists an English source (x₂ = The … x₇ = policies) and a Tamil target — i.e., English→Tamil. Figure 5 has the reverse inconsistency. Captions or mappings need swapping.","section":"§4.3.2, Figures 4–5"},{"comment":"Dataset names are inconsistently spelled throughout: 'EnTamV2'/'EntamV2'/'En TamV2', 'NTREX'/'NTrex'/'NTrex Benchmark', 'WikiMatrix'/'Wiki Matrix'/'WikiMatri x' (Table 2), 'TamilLaMA'/'Tamil LaMA'/'Tamil Lama'. Please standardize.","section":"Throughout"},{"comment":"§4.1.2 claims decoding optimization yielded 'increases of around 0.5–1.0 BLEU points and a chrF score between 1 and 2 points,' but no before/after numbers are given. Either include a small ablation table (baseline decoding vs tuned decoding) or remove the quantitative claim.","section":"§4.1.2"},{"comment":"No statistical reliability information is provided: single decoding run per configuration, no confidence intervals or significance tests (e.g., bootstrap resampling for BLEU/chrF), and test-set sizes per dataset are never stated. At minimum, report test-set sizes and sacreBLEU confidence intervals.","section":"§3.4; §4.1"},{"comment":"The heatmaps in Figures 2–6 are described qualitatively ('clear and nearly diagonal', 'fuzzy and irregular') but the figures are small and unlabeled in the preprint; axis labels (token identities), colorbars, and the averaging scheme over layers/heads should be specified. Note also that averaging or selecting the 'final layer' (§4.3.1) materially changes attention patterns; justify the choice.","section":"Figures 2–6; §4.3.1"},{"comment":"chrF is described as measuring 'semantic similarity at a character level' (§3.4) — it measures character n-gram overlap, not semantics. Also, chrF version (chrF vs chrF++) and the BLEU tokenization should be stated for reproducibility.","section":"§3.4"},{"comment":"Several references are incomplete or missing venue/year details: Thillainathan et al. (2025), Raja & Vats (2025), Lakew et al. (2018), Hendy et al. (2023), Jiao et al. (2023) lack full bibliographic data; Gangadharaiah et al. (2023) lacks a venue. The NTREX dataset itself is used but never cited (Rikters et al. / the Microsoft NTREX-128 release).","section":"References"},{"comment":"Typos and grammar: 'explain ability' (Abstract); 'This performs an attention-based analysis' (Abstract, missing subject); 'multi NMT models' (§1.1); 'the behavior of the translates' (§4.3); 'rep resents' (§4.3.3); 'Tamil capable TamilLaMA' (Abstract). A careful proofreading pass is warranted.","section":"Throughout"},{"comment":"The qualitative examples in §4.2.2 are used to claim TamilLaMA is 'more fluent and natural,' but with two cherry-picked sentences and no annotator protocol this cannot support even a qualitative conclusion. Either add a small structured human evaluation (raters, criteria, agreement) or soften the claim to an illustrative observation.","section":"§4.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has the shape of a systematic study but several numbers in Table 2 (the PMIndia collapse, the duplicated 9.56 PB-SMT baseline, the mBART WikiMatrix outlier) look like un-diagnosed pipeline or reporting artifacts rather than measured phenomena. The visible Unicode corruption in the quoted Tamil output reinforces this. If the authors can produce a clean re-run with disclosed checkpoints and preprocessing, the dataset-quality comparison could be a modest but useful contribution; as submitted, the central evidence does not survive scrutiny. Fit with the journal is reasonable, but I would want the re-run before any acceptance."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: they ran off-the-shelf NLLB and mBART both ways on four public English–Tamil sets and tied the score gaps to data quality, with attention entropy as a supporting diagnostic and a small TamilLaMA few-shot contrast. That specific measurement set is new even if the template is not.\n\nWhat works. EnTamV2 vs NTREX behaves as you’d expect—cleaner alignment, better BLEU/chrF—and reporting both metrics for a morphologically rich pair is the right call. The attention heatmaps and entropy numbers give a concrete, if standard, way to show that fuzzy cross-attention tracks the noisy sets. The qualitative LLaMA examples are honest about fluency vs literal accuracy. Limitations section is short but not evasive.\n\nSoft spots, in proportion. The load-bearing PMIndia row (BLEU ~0.25, chrF ~0.5) is not credible as “very high noise.” chrF that low usually means empty output, wrong language, ID misalignment, or encoding death—not domain mismatch on human-translated government text. Their own cited PB-SMT baseline got ~9.5 BLEU on PMIndia, and the Tamil strings printed in the paper are visibly corrupted. That undercuts the strongest evidence for the central claim. WikiMatrix is the other anomaly (mBART EN→TA BLEU 28 vs NLLB ~10 on the same data; BLEU far above the “clean” sets) and is waved off as inflated without error analysis. Everything is inference-only with unspecified beam/length settings, no seeds, no significance, no released configs—so attributing gaps purely to dataset noise rather than pretrain-domain fit is weaker than the conclusion states. Novelty is incremental; Table 1 already lists the nearby Indic/LLM MT work.\n\nWho it’s for: people who actually pick English–Tamil corpora and want a cautionary multi-set snapshot. Math is just standard metrics plus entropy; citations are fine; data story is the point and it’s half-broken on the dramatic end.\n\nI’d send it to peer review at a workshop or specialized track expecting major revision (fix the eval pipeline, release checkpoints/decoding, same-test-set LLM numbers, tone down “comprehensive”). I would not prioritize it for a top-venue reading group or cite the PMIndia numbers as-is.","headline":"Useful English–Tamil multi-dataset bake-off, but the PMIndia near-zeros look like a broken eval pipeline, not clean proof that noise dominates.","tokens_in":10949,"tokens_out":584,"would_cite":false,"duration_ms":24814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"For English–Tamil translation, parallel-data quality and domain alignment dominate model scores more than which off-the-shelf multilingual system you pick.","keywords":["English-Tamil machine translation","low-resource NMT","NLLB","mBART","attention entropy","dataset quality","few-shot LLM translation","chrF"],"falsifier":"Fine-tune NLLB and mBART on equal amounts of cleaned EnTamV2-style versus PMIndia-style data under the same recipe, then check whether the huge metric gap shrinks or vanishes; if it vanishes, dataset quality is not the dominant driver under the paper’s setup.","tokens_in":10584,"feed_emoji":"🌐","tokens_out":906,"duration_ms":16162,"temperature":0.7,"pith_summary":"English–Tamil and Tamil–English remain hard for neural machine translation because parallel data are scarce, often noisy, and domains shift, while Tamil’s morphology is rich. This paper runs two widely used multilingual transformers, NLLB and mBART, off the shelf on four parallel sets that range from clean (EnTamV2) to severely mismatched (PMIndia), scoring them with BLEU and chrF and reading their cross-attention. It finds that well-aligned data give stable, moderate gains and that very noisy data drive scores near zero, while lower attention entropy tracks better quality. A Tamil-adapted LLaMA used few-shot also produces fluent conversational output, though it is less literal than the supervised systems on harder sentences. A sympathetic reader cares because the result says where to spend effort first: clean and match the data before chasing larger models.","feed_headline":"Data quality beats model choice for English–Tamil MT","feed_subtitle":"Clean parallel sets lift NLLB and mBART; noisy PMIndia scores near zero, and attention entropy tracks the gap.","key_machinery":"Cross-attention heatmaps and sentence-level attention entropy H(y) = −∑ aᵢⱼ log(aᵢⱼ) on the decoder’s final layer, used as a quantitative proxy for how sharply source tokens align to target tokens and how that sharpness tracks BLEU/chrF across clean versus noisy corpora.","core_discovery":"Across English–Tamil and Tamil–English, dataset noise level and alignment quality largely determine the BLEU and chrF of off-the-shelf NLLB and mBART: clean, well-aligned sets such as EnTamV2 yield consistent moderate scores, while highly noisy, domain-mismatched sets such as PMIndia yield near-zero scores, and lower cross-attention entropy co-occurs with better automatic quality. Few-shot TamilLaMA can still produce structurally coherent, fluent translations, especially in conversational settings.","pith_inferences":["Corpus-quality gates and automatic noise filters may be higher-leverage investments for Indic MT than scaling another generic multilingual model.","The same entropy-versus-quality pattern could be tested as a cheap filter when mining new web parallel data for other agglutinative low-resource languages.","Hybrid pipelines that use supervised NMT for literal fidelity and few-shot LLMs for conversational polish are a natural next experiment the paper leaves open."],"forward_implications":["Cleaning and domain-matching English–Tamil parallel data should raise BLEU/chrF more reliably than swapping among off-the-shelf multilingual checkpoints alone.","Attention entropy can serve as a cheap diagnostic for whether a corpus is too noisy or misaligned before large training runs.","chrF should be reported alongside BLEU for Tamil, because inflated BLEU with unstable chrF flags misalignment (as on WikiMatrix).","Few-shot Tamil-capable LLMs are usable for fluent conversational translation but remain weaker than supervised NMT on literal, entity-heavy, or long sentences."],"fun_headline_variants":["Dataset noise swamps model choice in English–Tamil MT","Clean sets lift NLLB mBART; noisy PMIndia scores near zero","Attention entropy tracks English–Tamil quality gaps","Few-shot TamilLaMA still yields coherent Tamil translations","Domain match and alignment drive English–Tamil BLEU"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Score gaps can be blamed mainly on dataset noise and alignment when the models are only run with decoding tweaks and never fully fine-tuned or trained under matched conditions on each corpus.","fun_headline_variants_meta":{"raw":{"variants":["Dataset noise swamps model choice in English–Tamil MT","Clean sets lift NLLB mBART; noisy PMIndia scores near zero","Attention entropy tracks English–Tamil quality gaps","Few-shot TamilLaMA still yields coherent Tamil translations","Domain match and alignment drive English–Tamil BLEU"]},"model":"grok-4.5","effort":"low","cost_usd":0.004882,"raw_usage":{"total_tokens":1404,"prompt_tokens":831,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":48824000,"prompt_tokens_details":{"text_tokens":831,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":508,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":831,"tokens_out":65,"duration_ms":8849,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T12:40:28.925944+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Fine-tune NLLB and mBART on equal amounts of cleaned EnTamV2-style versus PMIndia-style data under the same recipe, then check whether the huge metric gap shrinks or vanishes; if it vanishes, dataset quality is not the dominant driver under the paper’s setup.","supporting_citations":[],"review_version":1}