{"id":"5b2bc309-c477-407c-bd7d-c53c7517a486","arxiv_id":"2412.20223","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 16-language African headline generation benchmark shows that the compact AfriTeVa V2 model matches the much larger Aya-101 LLM.","lead":"This paper introduces AfriHG, a benchmark dataset for generating news headlines in 16 African languages, built by merging the XLSum and MasakhaNEWS corpora. The authors show that a small Africa-focused model (AfriTeVa V2, 313M parameters) matches the performance of a 13B-parameter general-purpose LLM (Aya-101) on this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aya-101 was instruction-tuned on XL-SUM headline generation, so the central claim that 313M AfriTeVa is competitive with 'prompting' Aya-101 is confounded by Aya having already seen the same task and data; the paper itself states Aya 'has seen the same training data'.","rationale":"The reader identified the dataset construction for Shona/Xhosa/Zulu as the weakest assumption, but that concern affects only 3 of 16 languages and the aggregated averages still shift somewhat. A more fundamental issue is the Aya baseline's provenance: the paper explicitly states Aya was instruction-tuned on XL-SUM headline generation and 'has seen the same training data.' This makes the central comparison one of fine-tuned models on both sides, not fine-tuning vs prompting, and it means the abstract's claim as stated is not supported by the table. The concrete test would settle it: check Aya's training data for XL-Sum instances. If contamination is found, the paper's central claim is invalidated, though the AfriHG dataset can still be a useful resource with corrected claims. Because this can be addressed by re-evaluating with a clean LLM or by transparently reporting the contamination, the appropriate disposition remains conditional acceptance with major revisions, not outright rejection. The reader's conditional verdict is therefore directionally correct, but the condition set must be updated to include training-data provenance and overlap checks.","tokens_in":6581,"tokens_out":11380,"duration_ms":99196,"concrete_test":"Inspect the Aya Dataset (Singh et al., 2024) and the Aya-101 training recipe for inclusion of XL-Sum or AfriHG instances; fuzzy-match a sample of AfriHG test headlines against Aya's training examples. If any overlap exists, or if the dataset card confirms instruction tuning on XL-Sum headline generation, rerun the Aya evaluation with a 13B model that has not been trained on XL-Sum (e.g., a decoder-only model or mT5-XXL without task-specific instruction tuning) under the same prompt, and compare per-language ROUGE. If Aya's scores drop materially when the XL-Sum contamination is removed, the 'competitive' claim is an artifact of the confound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (abstract; §4) is that fine-tuned AfriTeVa V2 (313M) is competitive with 'prompting' Aya-101 (13B). The comparison is confounded by training contamination on Aya's side. In §1 the authors describe Aya as 'an instruction-tuned mT5-XXL model on several tasks including summarization and headline generation of XL-SUM (Hasan et al., 2021)'. AfriHG's 13 of 16 languages are built from XL-Sum (BBC article-headline pairs; §2). In §4 the paper concedes: 'While Aya LLM has a more than 43 times more parameters than AfriTeVa V2, and has seen the same training data, the performance are very similar.' If Aya was instruction-tuned on XL-Sum headline data—or on AfriHG's training split—then 'prompting Aya' is not a zero-shot/few-shot prompt baseline; it is an instruction-tuned model already fine-tuned on the target task and source corpus. The reported average R1 difference (AfriTeVa 28.5 vs Aya 28.2) is then unsurprising and does not support the claimed advantage of fine-tuning over prompting. The paper never reports the prompt template, decoding parameters, or whether any XL-Sum instances appear in Aya's training data, so the size of the confound is unquantified. This concern is more load-bearing than the dataset-construction issue for Shona/Xhosa/Zulu because it directly invalidates the paper's headline comparison rather than one subset's quality.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AfriHG, a news headline generation dataset for 16 African languages, built by combining article-headline pairs from XL-SUM with languages from the MasakhaNEWS corpus (Shona, isiXhosa, isiZulu). The authors fine-tune mT5-base and AfriTeVa V2-base and evaluate the instruction-tuned Aya-101 model, reporting ROUGE scores in Table 1. The main empirical claims are that AfriTeVa V2 outperforms mT5-base on average and that a 313M-parameter fine-tuned model is competitive with the 13B-parameter Aya-101 when the latter is prompted. The paper also claims that non-Latin script languages perform very poorly, with ROUGE-1 below 4.0, and releases code and data on GitHub.","tokens_in":6900,"tokens_out":6370,"duration_ms":57359,"significance":"If the comparisons were clean, the paper would offer a practically useful resource: a new multilingual headline generation benchmark for low-resource African languages, with reproducible fine-tuning recipes and a striking efficiency result (313M vs. 13B parameters). The dataset release and the use of held-out XL-SUM splits are concrete strengths, as is the systematic comparison across 16 languages. However, the central 'competitive to prompting Aya' claim is weakened by Aya's prior instruction-tuning on XL-SUM headline generation, which the paper itself acknowledges in Section 4. The contribution remains useful as a benchmark and a practical model comparison, but the headline claim needs to be reframed or supplemented with controls before it can be accepted as stated.","major_comments":[{"comment":"The central claim that fine-tuned AfriTeVa V2 (313M) is competitive with 'prompting' Aya-101 (13B) is confounded by Aya's prior exposure to the task and source corpus. Section 1 states that Aya was instruction-tuned on 'summarization and headline generation of XL-SUM,' and Section 4 concedes that Aya 'has seen the same training data.' Because 13 of the 16 AfriHG languages are built from XL-SUM BBC article-headline pairs, Aya is not a purely prompted baseline for this task. The paper does not report the prompt template, decoding parameters, or an analysis of whether any AfriHG test instances occur in Aya's instruction-tuning data. Without this information, the observed near-tied ROUGE averages in Table 1 (28.5 vs. 28.2) cannot support the claimed advantage of fine-tuning over prompting. Please quantify the overlap, add a baseline that has not been instruction-tuned on XL-SUM headline generation, or explicitly reframe the claim as 'competitive with an instruction-tuned model that has already seen the task.'","section":"Abstract; §4, Table 1"},{"comment":"The introduction's statement that 'languages with non-Latin script have a very low (<4.0 ROUGE-1) score on news headline generation' is contradicted by Table 1, which reports ROUGE-1 of 17.9 for Amharic, 25.3 for Arabic, and 21.9 for Tigrinya under mT5-base, and similar values for AfriTeVa. Section 4's discussion of script-specific tokenizers and a '−20 points' difference suggests the <4.0 figure may refer to a different condition (the default tokenizer), but that condition is not reported in any table. Please either report the tokenizer-dependent results explicitly or correct the introduction to match Table 1.","section":"§1 vs. Table 1"},{"comment":"For Shona, isiXhosa, and isiZulu, the dataset construction is underspecified. Section 2 states only that these languages come from the MasakhaNEWS corpus (VOA and Isolezwe sources), but MasakhaNEWS is a news topic classification dataset and does not, by itself, supply article-headline pairs. The paper does not describe how headlines were extracted or verified for these three languages. If the pairs are not genuine article-headline pairs, the per-language results in Table 1 and the aggregated averages are affected, and the comparison across 16 languages loses its basis. Please document the pairing procedure, include examples, and release the new pairs for these languages.","section":"§2"}],"minor_comments":[{"comment":"The statement that AfriTeVa was 'consistently better' than mT5 in all evaluated languages is contradicted by Arabic in Table 1 (mT5 R1=25.3 vs. AfriTeVa 24.1); revise to 'in 15 of 16 languages' or qualify the claim.","section":"§4"},{"comment":"Section 3 says Aya covers 12 of the evaluated languages, but Table 1 lists Aya scores for all 16 languages; clarify how scores were obtained for languages outside Aya's coverage.","section":"§3"},{"comment":"Please report the exact prompt template used for Aya, the decoding hyperparameters (beam size, max length, repetition penalty), and the ROUGE implementation and version, none of which are currently specified.","section":"§4"},{"comment":"The ratio 'more than 43 times' is arithmetically inaccurate: 13B / 313M is approximately 41.5; use 'more than 41 times' or compute from the exact parameter counts.","section":"§3"},{"comment":"Typos and formatting issues include 'focues' (should be 'focuses'), 'seq2eq' in the abstract, and irregular spacing in 'Y oruba' in Table 1; a careful proofread is needed.","section":"§1, Abstract"},{"comment":"Please specify how duplicates were removed after merging XL-SUM and MasakhaNEWS and whether the 80/10/10 split for the three non-XL-SUM languages was applied at the article level or the headline level, to avoid potential overlap across splits.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"This is a useful workshop-level benchmark paper, and the data release is a tangible contribution. However, the abstract overstates the Aya comparison, and the <4.0 ROUGE claim in the introduction contradicts Table 1. These issues should be fixed before publication; the paper may be acceptable after the requested revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the resource: AfriHG merges XL-Sum and MasakhaNEWS into a 16-language headline generation benchmark, and the authors release code and data. That is genuinely useful for low-resource NLP, and the paper should be read as a benchmark contribution, not a methodological advance. Second, the central comparison in the abstract is not clean. AfriTeVa V2 (313M) fine-tuned indeed scores about the same as Aya-101 (13B) on average ROUGE (28.5 vs 28.2), but the authors themselves note that Aya 'has seen the same training data.' Since 13 of the 16 languages come from XL-Sum, and Aya was instruction-tuned on XL-Sum headline generation, prompting Aya is not a zero-shot baseline; it is a model already fine-tuned on the target task and source corpus. That confound directly weakens the claim that fine-tuning small models is competitive with prompting large ones. The stress-test note is right, and it is more serious than the dataset-construction questions.\n\nWhat the paper does well: the mT5 vs AfriTeVa comparison is straightforward and supports the unsurprising conclusion that an Africa-focused pretrained model beats a general multilingual one. The authors are also honest about the Aya contamination, which is more than many papers manage.\n\nSoft spots, in order. (1) The Aya comparison needs reframing or a proper zero-shot baseline (e.g., a model not trained on XL-Sum). (2) The introduction says non-Latin script R1 is below 4.0, but Table 1 reports 17.9–26.3 for Amharic, Arabic, and Tigrinya. The paper later explains this is about tokenizer choice, but the abstract and introduction are misleading and should be corrected. (3) The construction for Shona, isiXhosa, and isiZulu is under-specified; no detail on how headlines were obtained from Isolezwe and VOA. (4) No error bars or significance tests; for a small benchmark paper this is minor but worth adding.\n\nWho is this for? Anyone building or evaluating models for African-language summarization or headline generation. It deserves a serious referee: the dataset is real, the code and data are promised, and the flaws are fixable with a revised comparison and clearer writing. Not a breakthrough, but a legitimate data point.","headline":"Useful merged benchmark for African headline generation, but the central AfriTeVa-vs-Aya claim is undercut by the paper's own admission that Aya was trained on the same data.","tokens_in":7497,"tokens_out":2676,"would_cite":true,"duration_ms":23995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 313M-parameter African-language model matches a 13B instruction-tuned LLM on news headline generation.","keywords":["headline generation","African languages","abstractive summarization","AfriTeVa V2","mT5","Aya-101","ROUGE","low-resource NLP"],"falsifier":"Take a random sample of AfriHG test pairs for Shona, isiXhosa, and isiZulu and check each 'headline' against the original VOA or Isolezwe article page; if many are section labels, captions, or editorially rewritten titles, then the ROUGE scores for those languages do not measure headline generation and the reported averages would need to be recomputed.","tokens_in":6363,"feed_emoji":"📰","tokens_out":8158,"duration_ms":67615,"temperature":0.7,"pith_summary":"The paper introduces AfriHG, a news headline generation dataset for 16 African languages, assembled from XL-Sum BBC articles and the MasakhaNEWS corpus. It fine-tunes two sequence-to-sequence models, mT5-base and AfriTeVa V2-base, and compares them with prompting Aya-101, a 13B instruction-tuned multilingual model. The central claim is that the Africa-centric AfriTeVa V2 outperforms mT5-base and is competitive with Aya-101 despite having 313M parameters versus 13B, with average ROUGE-1 scores of 28.5 versus 28.2. If true, this makes fine-tuned compact models a practical alternative to very large prompted LLMs for low-resource African headline generation.","feed_headline":"Compact African model matches a 13B LLM on news headlines","feed_subtitle":"Fine-tuned AfriTeVa V2 scores 28.5 ROUGE-1 versus 28.2 for the 13B Aya-101 across 16 languages.","key_machinery":"The central object is AfriHG, a dataset of news article-headline pairs covering 16 African languages, built by merging the XL-Sum dataset with the MasakhaNEWS corpus and adding Isolezwe and VOA sources for isiXhosa, isiZulu, and Shona. The comparison mechanism is headline generation framed as abstractive summarization, scored with ROUGE. The argument is carried by three models: mT5-base (580M parameters), AfriTeVa V2-base (313M), and Aya-101 (13B), the last prompted rather than fine-tuned.","core_discovery":"AfriTeVa V2-base, fine-tuned on AfriHG, achieves average ROUGE-1/2/L of 28.5/10.8/25.5 across 16 languages, beating mT5-base by +2.8/+1.5/+2.6 and nearly matching Aya-101's 28.2/10.9/24.9. Aya-101 posts the best score in 9 of the 16 languages, but the averages are almost identical. The authors interpret this as evidence that the fine-tuning paradigm remains competitive with prompting large instruction-tuned models when abundant training data exists, and that pre-training on more African languages is what gives AfriTeVa V2 its edge over mT5. The paper also reports that non-Latin-script languages such as Arabic, Amharic, and Tigrinya score far lower unless a script-specific tokenizer is used.","pith_inferences":["A practical extension the paper leaves implicit: if AfriTeVa V2's competitiveness holds under human evaluation, the cost and latency savings versus a 13B model could make automated headline generation feasible for smaller African news outlets.","The validity of the benchmark for Shona, isiXhosa, and isiZulu rests on the unstated assumption that the VOA and Isolezwe pages provide true article-headline pairs; spot-checking those pages would confirm or refute this.","ROUGE's n-gram overlap may punish valid paraphrased headlines, so a crowdsourced judgment study could change the apparent Aya-101 versus AfriTeVa V2 tie without contradicting the ROUGE numbers.","The reported tokenizer effect is testable directly: re-running the non-Latin languages with script-specific tokenizers should raise ROUGE by roughly 20 points if the explanation is right, and would be a low-cost follow-up experiment."],"forward_implications":["AfriHG gives the research community a 16-language benchmark for news headline generation, adding Shona, isiXhosa, and isiZulu to the languages already covered by XL-Sum.","Fine-tuned AfriTeVa V2 achieves headline quality comparable to a prompted 13B model, making the fine-tuning route a practical option for African news summarization systems with limited compute.","The reported roughly 20-point ROUGE drop for Arabic, Amharic, and Tigrinya with default tokenizers indicates that script-aware tokenization is a separate and important factor for non-Latin African languages.","Because Aya-101 was trained on the same underlying data at much larger scale, the near-tie in ROUGE supports the paper's conclusion that task-specific fine-tuning still has an advantage when abundant training data is available."],"supporting_citations":[{"why":"Supplies the BBC article-headline pairs for 13 of the languages and the baseline summarization scores used in Table 1.","marker":"Hasan et al., 2021"},{"why":"Provides the MasakhaNEWS corpus, the source of the Shona, isiXhosa, and isiZulu article-headline data.","marker":"Adelani et al., 2023"},{"why":"Defines mT5-base, the massively multilingual baseline model that AfriTeVa V2 is shown to outperform.","marker":"Xue et al., 2021"},{"why":"Introduces AfriTeVa V2, the central Africa-centric model fine-tuned in the headline generation experiments.","marker":"Oladipo et al., 2023"},{"why":"Introduces Aya-101, the 13B instruction-tuned model used for the prompted comparison.","marker":"Üstün et al., 2024"},{"why":"Defines ROUGE, the evaluation metric used for all headline generation comparisons.","marker":"Lin, 2004"},{"why":"Provides the T5 sequence-to-sequence architecture underlying both fine-tuned models.","marker":"Raffel et al., 2020"},{"why":"Introduces the original AfriTeVa model, the predecessor whose approach AfriTeVa V2 extends to more African languages.","marker":"Ogundepo et al., 2022"}],"fun_headline_variants":["Fine-tuned 313M model rivals 13B LLM on African headlines","African-focused model tops mT5, ties with 13B Aya on news","Small model matches giant LLM for African news headlines","AfriTeVa V2: 313M params, 28.5 ROUGE, beats mT5-base","Why a compact African model keeps pace with a 13B LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that the Shona, isiXhosa, and isiZulu article-headline pairs assembled from VOA and Isolezwe are genuine published headlines, though the paper does not describe how those pairs were extracted or validated.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned 313M model rivals 13B LLM on African headlines","African-focused model tops mT5, ties with 13B Aya on news","Small model matches giant LLM for African news headlines","AfriTeVa V2: 313M params, 28.5 ROUGE, beats mT5-base","Why a compact African model keeps pace with a 13B LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000645,"raw_usage":{"total_tokens":2917,"prompt_tokens":848,"completion_tokens":2069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":1961}},"tokens_in":464,"tokens_out":2069,"duration_ms":13821,"temperature":1.0,"reasoning_tokens":1961,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:24:59.378549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of AfriHG test pairs for Shona, isiXhosa, and isiZulu and check each 'headline' against the original VOA or Isolezwe article page; if many are section labels, captions, or editorially rewritten titles, then the ROUGE scores for those languages do not measure headline generation and the reported averages would need to be recomputed.","supporting_citations":[{"cited_title":"Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M","cited_arxiv_id":null,"evidence_quote":"Supplies the BBC article-headline pairs for 13 of the languages and the baseline summarization scores used in Table 1."},{"cited_title":"ROUGE : A package for automatic evaluation of summaries","cited_arxiv_id":null,"evidence_quote":"Defines ROUGE, the evaluation metric used for all headline generation comparisons."}],"review_version":1}