{"id":"1fed7aee-924e-43a5-90ba-3271a91e9703","arxiv_id":"2501.15281","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On a roughly 11-million-token Sepedi corpus, standard autoregressive pre-training gives lower validation loss and perplexity, while occlusion-based pre-training gives a slightly higher BLEU score on generated text.","lead":"The authors built two small Sepedi text-generation models, one trained with standard next-word prediction and one trained with randomly masked tokens, then compared them on a radio news task. The standard model had lower perplexity, while the masked model scored higher on BLEU, giving the first reported Sepedi generation baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison is confounded: the non-occlusion model uses 8 transformer layers and 8 attention heads, while the occlusion model uses 6 layers and 4 heads (Table 4), so capacity differences alone could explain the reported validation loss and perplexity gap.","rationale":"The reader's weakest_assumption identifies the same unmatched-architecture confound, and I agree. The central claim depends entirely on comparing two models that differ in two ways simultaneously: pre-training objective and model capacity. The paper does not justify why the hyperparameter search selected different architectures, and no control experiment separates capacity from objective. Because the capacity difference favors the non-occlusion model on likelihood-based metrics, the headline result is overclaimed. The missing BLEU generation protocol compounds the issue, but the architecture mismatch alone is sufficient to invalidate the comparison as currently presented. Given the useful dataset contributions and the fact that the confound is fixable, the conditional verdict is appropriate: the paper should not be accepted as-is, but revision with matched models and full evaluation details could support the claim. No adjustment to the reader's verdict is needed.","tokens_in":11191,"tokens_out":3428,"duration_ms":32265,"concrete_test":"Retrain SepGPT and SepGPT-OCC with identical architecture (e.g., both 8 layers and 8 heads) and identical hyperparameters (batch size, learning rate, dropout, vocab size, weight decay, training steps), varying only the presence or absence of the occlusion objective, across at least 3 random seeds. Report mean and standard deviation of validation loss, perplexity, and BLEU using a fixed decoding protocol (e.g., greedy or fixed temperature, same prompt set, same reference set). If the validation loss/perplexity gap persists under matched capacity, the objective effect is supported; if it disappears or reverses, the Table 3 comparison was confounded by architecture.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's claim is that the pre-training objective (occlusion vs non-occlusion) determines which metric improves. Table 4 shows the two compared models differ not only in objective: non-occlusion SepGPT uses 8 transformer layers and 8 attention heads, while occlusion SepGPT-OCC uses 6 layers and 4 heads. The hyperparameter search independently optimized each model, so architecture and objective are entangled. On the same data, a larger model generally achieves lower validation loss and perplexity, so the reported gap (validation loss 2.78 vs 3.46; validation perplexity 16.04 vs 31.89) is exactly what capacity differences would predict and cannot be attributed to masking. The higher BLEU score for the smaller occlusion model is likewise uninterpretable without matched capacity, and the BLEU generation protocol (decoding method, temperature, prompt set, reference alignment, number of generated samples) is not described. No passage acknowledges this confound; the paper only warns that cross-language comparisons are not directly comparable. The datasets and Sepedi baseline models remain useful, but the comparative claim in the abstract is not supported by the current experiment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper curates two new Sepedi text datasets (SepMono, a mixed monolingual corpus, and SepNews, a radio-news corpus split temporally into SepNews-1 and SepNews-2), pre-trains two GPT-2-style transformer models on SepMono using a standard next-token objective and an occlusion/masking-augmented objective, and fine-tunes both on SepNews-1. Evaluation is reported via validation/test loss, perplexity, and BLEU scores against reference texts. The abstract claims that non-occlusion models achieve better validation loss and perplexity, while occlusion-based models achieve slightly higher BLEU scores on generated text. The paper also compares these models with several other low-resource transformer baselines and presents example generated text.","tokens_in":147,"tokens_out":3101,"duration_ms":56884,"significance":"If the comparison were valid, the paper would offer a useful empirical data point for pre-training objectives on very small corpora in a low-resource language: standard next-token prediction giving lower perplexity while masking-based pre-training giving higher BLEU overlap. The datasets themselves, the temporal split of the news corpus into fine-tuning and test partitions, and the reproducible fine-tuning protocol (gradual unfreezing, early stopping) are contributions that could support future Sepedi NLP work. However, the central comparative claim is currently not supported because the two models differ in architecture as well as objective; the paper itself does not acknowledge or control for this confound.","major_comments":[{"comment":"The central comparison is confounded. The non-occlusion SepGPT model uses 8 transformer layers and 8 attention heads, while the occlusion-based SepGPT-OCC model uses 6 layers and 4 heads (Table 4). Because the hyperparameters were optimized independently for each model, the reported differences in validation loss (2.78 vs 3.46) and validation perplexity (16.04 vs 31.89) cannot be attributed to the pre-training objective rather than to model capacity. The same confound invalidates the BLEU comparison. To support the abstract's claim, the authors need to either compare models that are matched in depth, width, and parameter count, or include an explicit ablation showing that the capacity difference does not drive the observed pattern.","section":"§3.2 'Model training' and Table 4"},{"comment":"The BLEU evaluation is described without a generation protocol. The paper does not state the decoding method (greedy, sampling, beam search), temperature, prompt set, number of generated texts, or how generated texts are aligned with reference texts. Consequently, the reported BLEU differences ('5% higher', '3.86% higher') are not reproducible or interpretable. The authors should specify the full generation setup and report variance or confidence intervals, ideally over multiple seeds or prompts.","section":"§3.3 'Evaluation metrics' and §4 'Results'"}],"minor_comments":[{"comment":"The table contains formatting and typographical errors: 'T rainingV alidationT esting' should be a single header, and the unique-token total '125,04' appears to be missing a digit; please correct these.","section":"Table 2"},{"comment":"The sentence 'We further generated text from the trained models and computed a BLEU score' would benefit from specifying which of the four models were used for generation and whether the BLEU score is computed on pre-trained, fine-tuned, or both model variants; as written, the following percentages are ambiguous.","section":"§4 'Results'"},{"comment":"The phrase 'the the BLEU score' contains a duplicated article; please proofread.","section":"§4 'Results'"},{"comment":"The comparison with Sudanese GPT-2, RoGPT-2, and BanglaGPT in Table 3 is acknowledged by the authors as not directly comparable because datasets and languages differ, but the conclusion that 'our models obtained higher and comparable results' relies on this table; I suggest softening that conclusion or moving the external comparison to a clearly labeled reference row.","section":"§2.2 'Closely related studies'"}],"recommendation":"major_revision","confidential_remarks":"The paper's resource contributions are real, but the central scientific claim about occlusion versus non-occlusion pre-training is entangled with architecture differences. If the authors can rerun at least one matched-capacity comparison (e.g., a non-occlusion model with the occlusion model's 6-layer/4-head configuration, or an occlusion model with the 8-layer/8-head configuration), the paper could become publishable as an empirical study. Without such an experiment, the abstract's claim would remain unsupported, and the paper should be repositioned as a dataset/resource paper with baseline models rather than a comparison of pre-training objectives."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper gives you two genuinely new Sepedi datasets (SepMono, SepNews) and the first reported generative baselines for Sepedi, with BLEU scores. That part is a real resource for the low-resource community. The second thing is that the paper's headline claim — occlusion vs non-occlusion pre-training trades off perplexity against BLEU — is not supported by the experiment as run, because the two models compared are not matched. From Table 4: the standard model uses 8 transformer layers and 8 attention heads; the occlusion model uses 6 and 4. So the reported validation loss/perplexity gap (2.78 vs 3.46 and 16.04 vs 31.89) is exactly what the capacity difference would predict. The BLEU advantage for the smaller occlusion model is uninterpretable without matching capacity.\n\nWhat the paper does well: careful data curation from multiple sources, a temporally disjoint test set (SepNews-2 from a later period), and appropriate caution that cross-language comparisons in Table 3 aren't directly comparable. The fine-tuning protocol is described, and the tokenizer adaptation is sensible.\n\nSoft spots, in order. First, the architecture confound above; the paper never acknowledges it, and the abstract states the comparison as if the only difference were the objective. Second, the BLEU evaluation lacks a protocol: no decoding method, temperature, prompt set, reference alignment, or number of samples, and no uncertainty estimates. Third, the \"5% higher\" phrasing is percentage points, not relative, and the test perplexities after fine-tuning (56.0 vs 56.7) are much higher than validation (14.87 vs 16.48); the paper attributes this to domain shift, plausible but under-analyzed.\n\nNone of this kills the dataset contribution. If the authors rerun with matched architectures, or reframe the claim as joint capacity-plus-objective comparison, the paper becomes solid. Even as is, it deserves a serious referee: the data and baselines justify the time. My recommendation: send it to review, but with a strong request to fix the comparison and report the generation protocol.","headline":"New Sepedi datasets and first generative baselines are worth having, but the central occlusion-vs-non-occlusion comparison is confounded by unmatched model architectures.","tokens_in":11962,"tokens_out":2933,"would_cite":true,"duration_ms":24841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For a small Sepedi corpus, the pre-training objective determines which metric improves: masking does better on BLEU, standard next-word training on perplexity.","keywords":["Transformer","Text generation","Pre-training","Occlusion-based training","Sepedi","Low-resourced languages","Perplexity","BLEU"],"falsifier":"Retrain SepGPT and SepGPT-OCC with identical architecture (same number of layers, attention heads, and all other hyperparameters) and compare validation loss, perplexity, and BLEU on the same SepNews-2 test set. If the gaps vanish or reverse, the original conclusion about objectives is not supported; if they persist, the small-scale finding is confirmed.","tokens_in":11029,"feed_emoji":"🗣️","tokens_out":8375,"duration_ms":60716,"temperature":0.7,"pith_summary":"The paper tests whether masking input tokens during pre-training helps a text-generation model when the available corpus is small. It builds two GPT-2-style models for Sepedi, one trained with standard next-word prediction and one that also randomly occludes tokens, then fine-tunes both on radio-news data. On validation loss and perplexity the non-occlusion model comes out ahead, while the occlusion model scores higher BLEU against reference text. The authors take this as evidence that the choice of pre-training objective determines which quality metric improves in a low-resource setting.","feed_headline":"Token masking lifts BLEU but not perplexity in a small Sepedi model","feed_subtitle":"Standard next-word training gives lower perplexity; masking yields higher BLEU and new Sepedi baselines.","key_machinery":"The central mechanism is the occlusion-based pre-training objective layered onto the GPT-2 decoder. In this variant, a hyperparameter-tuned fraction (0.3) of the input tokens is randomly replaced with a mask, and the model must recover those tokens from the surrounding context while still learning to predict the next token left-to-right. The paper compares this against the pure next-word objective, using validation loss, perplexity, and BLEU as the yardsticks, and attributes the divergent behaviour of the models to the bidirectional context that masking introduces.","core_discovery":"On the SepMono corpus (432,970 sentences, roughly 11.4 million tokens), the authors pre-train SepGPT with the standard autoregressive GPT-2 objective and SepGPT-OCC with the same objective plus random token occlusion at probability 0.3. The non-occlusion model reaches a validation perplexity of 16.04 versus 31.89 for the occlusion model, with validation losses of 2.78 and 3.46. After fine-tuning on SepNews-1 and testing on a held-out time period (SepNews-2), the perplexity gap narrows to 14.87 versus 16.48, but the occlusion model produces the higher BLEU score: 48.84% versus 44.98% for the fine-tuned models, and 29.48% versus 24.19% before fine-tuning. The authors interpret this as showing that occlusion-based pre-training makes the generated text more similar to reference text, even though it yields a less confident probability model by perplexity.","pith_inferences":["If the architecture mismatch in Table 4 is corrected, the BLEU advantage of occlusion might shrink or disappear, so the headline claim should be re-tested with matched model sizes.","The opposite ordering of perplexity and BLEU suggests that evaluating generation quality by perplexity alone can be misleading for low-resource masked pre-training.","The same two-objective comparison could be run on other low-resourced languages to see whether the pattern generalizes beyond Sepedi.","The authors' qualitative note that generated sentences are 'mostly grammatically correct' should be verified with human evaluation before being relied upon."],"forward_implications":["For a small Sepedi corpus, occlusion-based pre-training yields higher BLEU scores than standard next-word pre-training, both before and after fine-tuning.","Non-occlusion pre-training gives lower validation loss and perplexity, which the authors describe as a more reliable probability model.","Fine-tuning reduces the validation-perplexity gap between the two approaches from a factor of about two to a small margin.","The newly curated SepMono and SepNews datasets provide baselines and resources for future Sepedi and low-resource language modeling.","The reported BLEU scores (44.98% and 48.84% after fine-tuning) set a new benchmark for Sepedi text generation."],"supporting_citations":[{"why":"Supplies the GPT-2 decoder architecture and the standard next-word prediction objective used for the non-occlusion model.","marker":"[24]"},{"why":"Defines the masked-language-model (occlusion) objective that the paper adds during occlusion-based pre-training.","marker":"[6]"},{"why":"Provides evidence that combining occlusion with language modeling is effective, motivating the occlusion setup.","marker":"[10]"},{"why":"Introduces the transformer architecture that both Sepedi models are built on.","marker":"[31]"},{"why":"Introduces gradual unfreezing, the fine-tuning technique used to adapt the models to the radio-news domain.","marker":"[11]"},{"why":"One of the component corpora of the SepMono pre-training dataset.","marker":"[23]"}],"fun_headline_variants":["Masked pretraining boosts BLEU, hurts perplexity on Sepedi","Occlusion pretraining: better BLEU, worse perplexity in Sepedi LM","Sepedi GPT: masking improves text quality but not confidence","Token masking trades perplexity for BLEU in low-resource Sepedi","For small Sepedi corpus, masking yields higher BLEU, lower perplexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The two models differ not only in pre-training objective but also in size: the non-occlusion model has 8 transformer layers and 8 attention heads, the occlusion model 6 layers and 4 heads, so the performance gaps could be caused by capacity rather than by the objective.","fun_headline_variants_meta":{"raw":{"variants":["Masked pretraining boosts BLEU, hurts perplexity on Sepedi","Occlusion pretraining: better BLEU, worse perplexity in Sepedi LM","Sepedi GPT: masking improves text quality but not confidence","Token masking trades perplexity for BLEU in low-resource Sepedi","For small Sepedi corpus, masking yields higher BLEU, lower perplexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3397,"prompt_tokens":961,"completion_tokens":2436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2335}},"tokens_in":577,"tokens_out":2436,"duration_ms":13101,"temperature":1.0,"reasoning_tokens":2335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:26:12.979002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain SepGPT and SepGPT-OCC with identical architecture (same number of layers, attention heads, and all other hyperparameters) and compare validation loss, perplexity, and BLEU on the same SepNews-2 test set. If the gaps vanish or reverse, the original conclusion about objectives is not supported; if they persist, the small-scale finding is confirmed.","supporting_citations":[{"cited_title":"openAI blog1(8), 9 (Feb 2019)","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-2 decoder architecture and the standard next-word prediction objective used for the non-occlusion model."},{"cited_title":"In: Proceedings of the 2019 Conference of the North","cited_arxiv_id":null,"evidence_quote":"Defines the masked-language-model (occlusion) objective that the paper adds during occlusion-based pre-training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that combining occlusion with language modeling is effective, motivating the occlusion setup."},{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Introduces the transformer architecture that both Sepedi models are built on."},{"cited_title":"In: Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers)","cited_arxiv_id":null,"evidence_quote":"Introduces gradual unfreezing, the fine-tuning technique used to adapt the models to the radio-news domain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the component corpora of the SepMono pre-training dataset."}],"review_version":1}