{"id":"c05d2d14-dbbc-4ac2-87b8-19fb40747d13","arxiv_id":"2504.17252","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Applying known RNN and transfer-learning methods to English-Igbo yields modest BLEU scores, but the claimed +4.83 BLEU improvement over baselines is inconsistent with the paper's own tables.","lead":"The paper trains standard recurrent neural network and fine-tuned MarianNMT models for English-to-Igbo translation, reporting BLEU scores up to 0.43 on a small test set. The headline claim of a +4.83 BLEU gain over existing benchmarks is not supported by the paper's own tables, so the main quantitative claims should be treated with caution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +4.83 BLEU claim is unsupported: no train/test split is given for the 597-pair MarianNMT fine-tune, and 4.83 equals 0.4300−0.3817 (the authors' own RNN model), not the gain over the cited HuggingFace baselines.","rationale":"The reader correctly identified the missing train/test split as a load-bearing weakness, and I agree that the comparison to HuggingFace baselines is confounded by different test sets. My stress-test adds a more direct internal problem: the advertised +4.83 BLEU gain is not the difference between the transfer model and either benchmark listed in the paper. It is exactly the difference between the transfer model and the authors' own final RNN model. This means the headline claim is not merely unverified; it is contradicted by the paper's own table. The split issue could theoretically be fixed by releasing code and data, and the baseline issue could be fixed by correcting the scale and computing BLEU on a common test set, but as written the central result is not supported. I therefore recommend REJECT rather than CONDITIONAL, because the current manuscript does not provide enough information to salvage the specific '+4.83 over baselines' claim. I credit the paper for a clear low-resource framing and for reporting training-time and loss details, but those do not compensate for an unverifiable central comparison.","tokens_in":18588,"tokens_out":5055,"duration_ms":44580,"concrete_test":"Ask the authors for the exact 597-pair train/test split and the HuggingFace evaluation script, then recompute BLEU on the held-out test subset only. If test BLEU falls to or below 0.395, the claimed advantage over JW300 disappears; separately recompute the deltas from Table 13 (0.4300−0.395=0.035, 0.4300−0.380=0.050) to check whether any baseline yields +4.83.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that fine-tuning MarianNMT on 597 English–Igbo pairs yields BLEU 0.43, a '+4.83 BLEU point' gain over HuggingFace benchmarks. This claim fails on two internal grounds. First, Section 5.6 states the model was trained on 597 sentence pairs, and Figure 12 reports a BLEU distribution 'across 597 test samples', but no train/validation/test split is ever specified. If the 597 training pairs are also the 597 evaluation samples, the 0.43 score is inflated by memorization and cannot be compared to any external benchmark. Second, even taking Table 13 at face value, +4.83 is not the gain over HuggingFace baselines: Table 13 lists Transfer Learning BLEU 0.4300 versus Final Optimized RNN 0.3817, and 0.4300 − 0.3817 = 0.0483. Against the cited baselines the differences are 0.4300 − 0.395 = 0.035 (JW300) and 0.4300 − 0.380 = 0.050 (Tatoeba). Moreover, Table 11 reports JW300 BLEU as 39.5 and Tatoeba as 3.8, while Table 13 transcribes these as 0.395 and 0.380; the Tatoeba entry changes scale/decimal. Thus the central comparative claim is not reproducible from the manuscript's own numbers. The 70% 'semantic accuracy' is also only qualitatively supported by examples and is not a defined metric. The burden is on the authors to release the split and a corrected baseline comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RNN-based (LSTM/GRU) sequence-to-sequence models with attention, teacher forcing, and greedy/beam decoding for English-to-Igbo translation, and additionally fine-tunes a MarianNMT model through the SimpleTransformers framework. It reports a final RNN BLEU of 0.3817, a transfer-learning BLEU of 0.43 on 597 samples, a claimed improvement of +4.83 BLEU over HuggingFace baselines, and an estimated translation accuracy of 70%. The paper also reports English-French experiments that it claims surpass a Tatoeba benchmark.","tokens_in":19049,"tokens_out":9380,"duration_ms":85116,"significance":"If the empirical claims were reproducible, the transfer-learning result on a very small parallel corpus would be a useful data point for low-resource NMT for African languages. The paper also includes useful comparisons of attention scoring functions, decoding strategies, and hyperparameter choices, and the qualitative translation examples give some insight into model behavior. However, the central claims are not supported by the manuscript's own tables and descriptions: the evaluation split for the transfer-learning experiment is undocumented, the +4.83 BLEU figure is arithmetically inconsistent with the cited baselines, and the 70% accuracy is undefined. These are not cosmetic issues; they affect the abstract's main contribution and make the reported performance gains unverifiable.","major_comments":[{"comment":"The transfer-learning evaluation has no documented train/validation/test split. The text states that MarianNMT was fine-tuned on 597 sentence pairs, and Figure 12 reports a BLEU distribution 'across 597 test samples' with no indication that the evaluation set is disjoint from the training set. If the same 597 pairs are used for both training and evaluation, the reported BLEU of 0.43 is inflated by memorization and cannot be compared with any external benchmark. The RNN experiments in Section 4.5, described as testing on 300 English-Igbo sentence pairs, have the same missing split information. This is load-bearing because the paper's headline claim depends entirely on a valid held-out evaluation.","section":"Section 5.6, Figure 12"},{"comment":"The claimed '+4.83 BLEU points' is not the difference between the transfer model and the HuggingFace baselines. Table 13 lists the transfer model at 0.4300 and the final optimized RNN at 0.3817, and 0.4300 - 0.3817 = 0.0483, which is the apparent source of the '4.83' figure. The differences against the cited baselines are 0.4300 - 0.380 = 0.050 and 0.4300 - 0.395 = 0.035 (or 5.0 and 3.5 if the scores are rescaled to 0-100). The claim in Section 7 that the improvement is over existing HuggingFace English-Igbo baselines is therefore not supported by the paper's own table.","section":"Abstract; Section 7; Table 13"},{"comment":"The baseline BLEU values are not consistently scaled across tables. Table 11 reports JW300.en.ig BLEU as 39.5 and Tatoeba.en.ig as 3.8, while Table 13 reports the same two baselines as 0.395 and 0.380. The Tatoeba value changes by an order of magnitude, and no explanation is given for the rescaling. Since the central comparison is against these baselines, the manuscript needs a single consistent scale and explicit statement of which test sets were used to produce each score.","section":"Table 11; Table 13"},{"comment":"The '70% translation accuracy' figure is not defined or measured. It appears in the abstract and Section 4.5 as translation quality or semantic accuracy, and Section 5.6.4 supports it only with a qualitative statement about a small number of examples. No metric, rubric, annotator agreement, or error-count procedure is provided. This claim should be removed or replaced with a defined evaluation.","section":"Section 4.5; Section 5.6.4"},{"comment":"The English-French experiment also lacks a documented evaluation split. The text says the model was trained on a 70,000-sentence subset of Tatoeba and reports BLEU scores up to 0.590, but it does not state whether the evaluation pairs were held out from training. In addition, Section 7 states that the model surpassed the English-French benchmark by 9 BLEU points, whereas the table difference is 0.085; the scaling inconsistency makes the claimed cross-lingual generalization unverifiable.","section":"Section 5.5; Section 7"}],"minor_comments":[{"comment":"The beam width for beam search is reported inconsistently: Section 4.4.2 says beam width = 2, while Section 5.4 says beam width = 5. Please reconcile these values.","section":"Section 4.4.2; Section 5.4"},{"comment":"The final hyperparameters are unclear: Section 4.3.1 states a batch size of 128 and 1024 LSTM units, while Figure 10 and the text report that batch size 32 and dropout 0.5 give the best performance. The final model configuration should be stated in one place.","section":"Section 4.3.1; Section 5.4"},{"comment":"The data source is attributed to reference [26], but the exact dataset version, license, and release are not specified. Please cite the actual data release and state any filtering performed after download.","section":"Section 4.1"},{"comment":"The contribution 'first high-performing translation system for the low-resource English-Igbo language pair' is an overclaim given that the paper itself compares against existing HuggingFace English-Igbo benchmarks; please temper this claim and cite recent Igbo NMT work.","section":"Section 1"}],"recommendation":"reject","confidential_remarks":"The central evaluation protocol is undocumented and the headline number is contradicted by the manuscript's own tables. These are not local presentation issues; the paper would need to be substantially reworked, with a documented train/test split and a corrected baseline comparison, before the main claims could be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the headline BLEU claim does not hold up. The +4.83 gain in the abstract is, per Table 13, 0.4300 − 0.3817, which is the difference between the MarianNMT fine-tune and the authors' own RNN. Against the cited HuggingFace baselines the differences are about +3.5 (JW300) and +5.0 (Tatoeba), not +4.83. The Tatoeba baseline itself is given as 3.8 in Table 11 and 0.380 in Table 13, so the scale is inconsistent. More importantly, Section 5.6 says the fine-tune was trained on 597 sentence pairs, and Figure 12 reports BLEU \"across 597 test samples,\" with no train/test split described anywhere. As written, the central comparison is not interpretable.\n\nWhat the paper does well: it is a readable, honest engineering write-up of a standard recipe—LSTM/GRU seq2seq with attention, plus one MarianNMT fine-tune—applied to English–Igbo. The hyperparameter comparisons (LSTM vs GRU, attention scoring functions, greedy vs beam search) are the kind of detail practitioners find useful, and the qualitative examples are illustrative. That is real value, but it is not a research novelty. The corpus and the OPUS-MT baseline are cited from Ezeani et al. and Helsinki-NLP, and the methods are textbook.\n\nSoft spots, in order. First, the missing split and the inconsistent numbers are load-bearing. Without a clean split, the 0.43 BLEU is unexplained, and the \"70% semantic accuracy\" is not defined as a metric anywhere—it appears as an abstract claim and as a qualitative gloss in Table 14. Second, the paper calls itself \"the first high-performing system\" while citing the same HuggingFace model it claims to beat; that claim should be dropped or reworded. Third, the Limitations section blames GPU memory, but the real limitation is evaluation design. No code, data, or split is released, so nothing is reproducible.\n\nWho is this for? Someone building a low-resource MT baseline might skim the hyperparameter and qualitative sections. As a research claim, it does not deserve referee time in its current state. I would desk-reject and invite a resubmission if the authors supply the split, correct the arithmetic, define the accuracy metric, and release the code and data.","headline":"The headline BLEU claim does not survive contact with the paper's own tables, and the fine-tune evaluation has no documented train/test split; the RNN case study itself is a competent but standard engineering write-up.","tokens_in":19573,"tokens_out":3027,"would_cite":false,"duration_ms":29092,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"English-to-Igbo machine translation improves by 4.83 BLEU points through transfer learning.","keywords":["English-to-Igbo translation","low-resource machine translation","recurrent neural networks","LSTM","GRU","attention mechanism","transfer learning","BLEU score"],"falsifier":"Evaluate the fine-tuned model on a held-out set of English-Igbo sentences that were never used in fine-tuning, and also run it on the exact test sets behind the published Tatoeba and JW300 baseline scores; if the score falls to near or below 0.395, the claimed +4.83 gain is an artifact of test-set overlap or score incomparability.","tokens_in":18399,"feed_emoji":"🌍","tokens_out":10133,"duration_ms":80502,"temperature":0.7,"pith_summary":"This paper targets English-to-Igbo translation, a low-resource pair for a language spoken by over 40 million people. It tries to show that a classical recurrent encoder-decoder with attention trained on roughly 12,000 verified sentence pairs can match existing benchmark quality, and that fine-tuning a pretrained MarianNMT model on a small set of English-Igbo pairs goes further, reaching a BLEU score of 0.43. That would be about 4.83 BLEU points above the published English-Igbo baselines, with roughly 70 percent of outputs judged semantically accurate. If correct, the result would offer a practical recipe for building usable translation systems for under-served languages with modest data and compute.","feed_headline":"Fine-tuned model lifts English-to-Igbo translation by 4.83 BLEU points","feed_subtitle":"A small attention-based RNN nearly matches existing benchmarks; fine-tuning a pretrained model goes further.","key_machinery":"The argument rides on two complementary systems. The first is an attention-equipped recurrent encoder-decoder: an LSTM encoder with 1024 units and 256-dimensional embeddings, a decoder using dot-product global attention, teacher forcing during training, and greedy decoding at inference. The second is MarianNMT, a pretrained encoder-decoder transformer for machine translation, fine-tuned for 20 epochs on 597 verified English-Igbo sentence pairs. The attention mechanism lets the decoder align output tokens to input tokens, which matters for low-resource pairs where the model cannot rely on massive data; the pretrained transformer supplies general translation knowledge that the small fine-tuning set then adapts to Igbo. Comparing the two systems against published English-Igbo benchmarks is what turns the numbers into the claimed +4.83 BLEU gain.","core_discovery":"The paper's central claim is that transfer learning closes the quality gap for English-to-Igbo machine translation. A from-scratch LSTM sequence-to-sequence model with dot-product global attention, teacher forcing, and greedy decoding reaches BLEU 0.3817, close to the published JW300 benchmark of 0.395. Fine-tuning the pretrained MarianNMT transformer for 20 epochs on 597 English-Igbo sentence pairs raises the score to 0.43, above both published baselines (Tatoeba 0.38 and JW300 0.395); the paper reports this as a gain of +4.83 BLEU points and an estimated 70% semantic translation accuracy. The same pipeline also surpasses the English-French Tatoeba benchmark by roughly 9 BLEU points, which the authors take as evidence the approach carries across language pairs.","pith_inferences":["The paper never states how the 597 transfer-learning pairs were split into training and test sets, so the first test of the +4.83 gain is whether the evaluation sentences were also used for fine-tuning.","The same recipe could be tried on other African language pairs: if a pretrained multilingual checkpoint exists, a few hundred expert-verified sentence pairs may be enough to beat from-scratch RNNs.","For a morphologically rich language like Igbo, BLEU is a weak proxy for quality, so the 70% semantic-accuracy estimate would be more convincing if confirmed by human ratings or a character-level metric."],"forward_implications":["Fine-tuning a pretrained MarianNMT model on 597 English-Igbo pairs yields BLEU 0.43, above the published Tatoeba (0.38) and JW300 (0.395) baselines.","An LSTM sequence-to-sequence model with dot-product attention reaches BLEU 0.3817 with a 16K vocabulary, essentially matching the JW300 baseline without pretraining.","The fine-tuned model produced semantically accurate translations on more than 70% of a 597-sample evaluation set, including long and special-character sentences.","The same architecture outperformed the English-French Tatoeba benchmark by about 9 BLEU points on a 70,000-pair subset, evidence for cross-lingual generalizability.","Since the fine-tuning loss was still decreasing at 20 epochs, the authors expect additional training to improve the score further."],"supporting_citations":[{"why":"Supplies the curated English-Igbo parallel corpus (Bible, news, Wikipedia, Common Crawl) that all models are trained on.","marker":"[26]"},{"why":"Provides the published English-Igbo benchmark BLEU scores (Tatoeba and JW300) that the +4.83 gain is measured against.","marker":"[34]"},{"why":"Defines the BLEU metric used for every quantitative comparison in the paper.","marker":"[54]"},{"why":"Provides the pretrained MarianNMT sequence-to-sequence implementation used for transfer-learning fine-tuning.","marker":"[59]"},{"why":"Establishes the transfer-learning-for-low-resource-NMT approach on which the fine-tuning strategy is built.","marker":"[78]"},{"why":"Supplies the attention mechanism that lets the RNN decoder align output tokens to encoder states.","marker":"[3]"},{"why":"Supplies the global attention scoring functions (dot-product, concat, general) compared in the experiments.","marker":"[46]"}],"fun_headline_variants":["Transfer learning lifts English-to-Igbo BLEU by 4.83 points","Low-resource Igbo translation boosted by transfer learning","Pretrained model beats Igbo baselines by 4.83 BLEU","English-Igbo NMT gains 4.83 BLEU with transfer learning","Fine-tuned MarianNMT improves Igbo translation by 4.83 BLEU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported +4.83 BLEU gain rests on the assumption that the 597 sentence pairs used in transfer learning are split into disjoint training and test sets, and that the resulting BLEU of 0.43 is directly comparable to benchmark scores computed on different test sentences.","fun_headline_variants_meta":{"raw":{"variants":["Transfer learning lifts English-to-Igbo BLEU by 4.83 points","Low-resource Igbo translation boosted by transfer learning","Pretrained model beats Igbo baselines by 4.83 BLEU","English-Igbo NMT gains 4.83 BLEU with transfer learning","Fine-tuned MarianNMT improves Igbo translation by 4.83 BLEU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1389,"prompt_tokens":926,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":542,"tokens_out":463,"duration_ms":3500,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:45:14.598813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the fine-tuned model on a held-out set of English-Igbo sentences that were never used in fine-tuning, and also run it on the exact test sets behind the published Tatoeba and JW300 baseline scores; if the score falls to near or below 0.395, the claimed +4.83 gain is an artifact of test-set overlap or score incomparability.","supporting_citations":[{"cited_title":"Igbo-English Machine Translation: An Evaluation Benchmark","cited_arxiv_id":"2004.00648","evidence_quote":"Supplies the curated English-Igbo parallel corpus (Bible, news, Wikipedia, Common Crawl) that all models are trained on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the published English-Igbo benchmark BLEU scores (Tatoeba and JW300) that the +4.83 gain is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the pretrained MarianNMT sequence-to-sequence implementation used for transfer-learning fine-tuning."}],"review_version":1}