{"id":"6be33cda-d699-4e25-8ecc-f085165494fb","arxiv_id":"2505.10081","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Probing shows African-language-adapted PLMs encode more syntactic and semantic information than massively multilingual PLMs, with word-level syntax concentrated in middle layers.","lead":"This paper uses probing, a method that reads what language models store in their inner layers, to analyze seven language models on six African languages. It finds that models adapted specifically for African languages encode more linguistic knowledge than general multilingual models, offering evidence for why such adaptations work.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"POS selectivity may be inflated for adapted models because the control task penalizes contextualized representations; the adapted-vs-multilingual advantage needs a control-accuracy check.","rationale":"The reader identified the standard probing critique and the weak NER baseline. I agree, but the most load-bearing version for the central claim is more specific: the MasakhaPOS control task may be systematically harder for contextualized representations, and since adapted models are likely more contextualized, the selectivity differences could reflect control-task difficulty rather than linguistic knowledge. This is a concrete, testable confound rather than a general philosophical objection. The paper is otherwise a careful application of established probing methods, and the control task design follows Hewitt and Liang (2019), so I do not think the paper should be rejected. It should remain conditional: the authors need to report control accuracies, add an alternative control condition, and provide variance estimates. My recommendation therefore does not change the reader's conditional verdict; it sharpens the specific test needed before the strongest claim can be relied upon.","tokens_in":12145,"tokens_out":8163,"duration_ms":91100,"concrete_test":"Recompute POS probing with two additions: (i) report control accuracy separately for each model and language; (ii) construct an occurrence-level control task in which every token is assigned an independent random label from the same POS distribution, removing word-type-to-label information, and recompute selectivity. If the adapted-vs-XLM-R ordering in Table 3 survives (for example, AfroXLMR-large still beats XLM-R-large on Swahili, Hausa, and isiXhosa), the original control is not driving the result. If the advantage shrinks or reverses, the central claim is an artifact of the word-type-constant control. Additionally, rerun with five control-mapping seeds and five NER re-initializations, reporting variance; the model ordering should be stable outside seed noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that adapted PLMs encode more linguistic information rests mainly on POS selectivity (Figure 3, Table 3). The control task in Section 3.2 gives each word type one random label, so a probe succeeds only if it maps every contextualized occurrence of a word type to the same arbitrary label. Models with more context-sensitive representations will find this harder, lowering control accuracy and mechanically raising selectivity. Adapted models such as AfroXLMR, Nguni-XLMR, and AfroLM are trained or adapted on target-language data and are likely more contextualized than XLM-R; if their control accuracy is systematically lower, the reported selectivity advantage does not establish additional linguistic knowledge. The paper never reports control accuracies separately, so this confound is untested. The NER gain over randomly re-initialized PLMs (Section 3.3) has a related issue: a non-contextualized random baseline can make any contextualized model look better regardless of NER-specific information. Because the headline conclusion and the causal statement in Section 5.2 depend on these comparisons, the confound must be resolved before the central claim is accepted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a probing study of seven multilingual PLMs across six African languages, using POS tagging, NER, and news topic classification. The authors design a control task for the MasakhaPOS dataset following Hewitt and Liang (2019), and use random baselines for NER and NTC. The central claim is that PLMs adapted for African languages encode more linguistic information about their target languages than massively multilingual PLMs, and that cross-lingual representation learning explains previously reported zero-shot NER performance. The paper reports layer-wise probe performance, best-layer results in Table 3, and qualitative trends about the layer distributions of syntactic and semantic information.","tokens_in":12337,"tokens_out":3320,"duration_ms":34128,"significance":"If the central claim holds, the paper would be a valuable contribution to interpretability research for African NLP, offering the first systematic probing analysis for these languages and introducing a reusable control task for MasakhaPOS. The broad coverage of languages, models, and tasks is a strength, and the layer-wise analysis connects to established findings in BERTology. However, the main comparative claim (adapted models encode more) rests on a metric that may be confounded by representation context-sensitivity, and the causal claim about zero-shot NER is not directly supported. The paper is promising but requires substantial additional analysis to substantiate its headline conclusion.","major_comments":[{"comment":"The headline claim that adapted PLMs encode more linguistic information is based primarily on POS selectivity, but control-task accuracies are never reported. In the Hewitt-Liang control task, each word type is assigned a fixed random label, so a probe must map all contextual occurrences of a word to the same arbitrary label. Models with more context-sensitive representations—which adapted models are likely to have—will find this harder, lowering control accuracy and mechanically raising selectivity even if the amount of task-relevant linguistic knowledge is unchanged. The paper should report control accuracies separately, and should verify the adapted-versus-multilingual advantage using a method that is not sensitive to this confound, for example by comparing raw accuracies conditioned on control accuracy or by using an alternative control that preserves contextual difficulty.","section":"§3.2, Fig. 3, Table 3"},{"comment":"The NER gain over randomly re-initialised PLMs is used to support both the encoding claim and the causal statement that cross-lingual representation learning is 'the mechanism behind' previously reported zero-shot MasakhaNER performance. A randomly re-initialised transformer is not a matched baseline for contextualisation: adapted models may show higher gains simply because their representations are more context-sensitive, not because they encode more NER-specific information. Moreover, probe gains are correlational evidence about representation content, not causal evidence about downstream task success. The causal statement in §5.2 should be removed or substantially softened.","section":"§3.3 and §5.2"},{"comment":"All results appear to come from a single run, with no error bars, multiple seeds, or significance tests. Many of the differences in Table 3 (e.g., POS selectivity 16.39 vs 19.09 for XLM-R-base vs XLM-R-large on Swahili) may be within run-to-run variability for MLP probes. The authors should provide variance estimates, at least for the best-layer comparisons that drive the central claim, or explicitly state that the reported differences are not statistically tested.","section":"§4 and Table 3"}],"minor_comments":[{"comment":"The y-axis label says 'F1-score' but the caption and text describe the plotted quantity as the gain over a random baseline; the label should read 'F1 gain' or similar.","section":"Fig. 4"},{"comment":"The phrase 'The six language in our study' should be 'The six languages in our study'.","section":"§4.2"},{"comment":"There is a typo: 'synactic' should be 'syntactic'.","section":"§5.1"},{"comment":"The sentence 'both exhibit high probe performance gains for Luganda than AfroLM' is ungrammatical; it should be 'both exhibit higher probe performance gains for Luganda than AfroLM'.","section":"§5.2"},{"comment":"For Igbo, several models show negative selectivity in some layers; this is mentioned in the text but not visualised clearly in the caption. Consider adding a note to the caption that negative values indicate control accuracy exceeding task accuracy.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an under-explored area and the control task design for MasakhaPOS is a useful contribution. However, the central empirical claim depends on a selectivity metric that is vulnerable to a contextualisation confound, and the causal language in §5.2 overreaches. The authors should be asked to provide control accuracies and a stress-test of the selectivity comparison before the central claim is accepted. The absence of variance estimates is also a concern for a journal-level acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one if you care about whether adaptation actually encodes linguistic knowledge, not just better downstream numbers. It is the first systematic probing study for African-language PLMs, and the MasakhaPOS control task is a real addition. But the headline claim needs one more check before I'd trust it: control accuracy.\n\nWhat is genuinely new: seven PLMs, six languages, three tasks. The probing framework is established, but the application is first, and the layer-wise findings for these languages are new even though they echo familiar BERTology patterns. The paper is also honest about its limitations, which I appreciate.\n\nThe biggest soft spot is the one the stress test flags. Selectivity equals task accuracy minus control accuracy. The control labels each word type with one random label, so a probe only succeeds if it maps every contextualized occurrence of that word type to the same label regardless of context. Models with more context-sensitive representations—exactly what adapted models should have—will get lower control accuracy, which mechanically raises selectivity. The paper never reports control accuracies separately. That is a straightforward missing table. If AfroXLMR, Nguni-XLMR, or AfroLM control accuracy is systematically lower than XLM-R's, part of the selectivity gap may be extraction difficulty rather than additional linguistic encoding. Raw accuracy in Figure 6 helps but does not settle it. I would not call this fatal: adaptation benefits are real and the raw accuracies look favorable. But the central claim is underdetermined by the reported metric.\n\nSecond, the NER baseline is weak. Randomly re-initializing the same architecture is common, but a non-contextualized baseline makes any contextualized model look good, so the gains in Figure 4 are probably inflated. The NTC baseline—an untrained bi-LSTM—is more defensible. Third, the causal sentence in Section 5.2, that cross-lingual representation learning is “the mechanism” behind MasakhaNER zero-shot results, goes beyond what probing can show; the data show correlation, not causation. Also, no error bars, no multiple seeds, and no code, so I cannot tell how stable the numbers are. All of these are fixable.\n\nWho this is for: people working on African NLP interpretability and anyone probing low-resource multilingual models. The control task for MasakhaPOS will likely be reused.\n\nRecommendation: send it to review. With control accuracies reported, variance estimates, code, and softened causal claims, it becomes a solid contribution. As written, it needs revision before publication.","headline":"First systematic probe of African-language PLMs; plausible and useful, but the selectivity confound and weak baselines need fixing before the central claim is accepted.","tokens_in":12840,"tokens_out":3059,"would_cite":false,"duration_ms":32737,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that language models adapted for African languages encode more linguistic information about those languages than massively multilingual models do, and that this encoded knowledge—not probe memorisation—drives their strong…","keywords":["probing","African languages","multilingual language models","control tasks","selectivity","part-of-speech tagging","named entity recognition","cross-lingual transfer"],"falsifier":"Permute the word identities used to build hidden representations by shuffling the token embedding table before feeding text through the model, then retrain the probe; if the measured selectivity does not drop sharply, the probe is recovering surface word-type statistics rather than the linguistic knowledge the paper attributes to the model.","tokens_in":11933,"feed_emoji":"🧠","tokens_out":8211,"duration_ms":75317,"temperature":0.7,"pith_summary":"The paper is the first systematic probing study of African-language pretrained models, covering six typologically diverse languages and three tasks. It claims that models adapted for African languages store more linguistic information about their target languages than massively multilingual models such as XLM-R, and that this is why adapted models perform well when transferred to new tasks. The authors design a control task for the MasakhaPOS dataset and use random baselines for named-entity recognition and topic classification to argue that the probe results reflect knowledge inside the model rather than memorisation by the probe. They also find syntactic information concentrated in middle-to-late layers while sentence-level semantic information is spread across all layers.","feed_headline":"Adapted African-language models encode more linguistic knowledge","feed_subtitle":"Probing six African languages shows adaptation—not scale—stores grammar and meaning in hidden layers.","key_machinery":"The load-bearing tools are probes and their baselines. Each probe is a one-hidden-layer neural network with 50 hidden units, trained on hidden representations to predict parts of speech, named-entity labels, or news topics. To distinguish model knowledge from probe learning, the paper builds a control task for MasakhaPOS by randomly mapping each word type to a part-of-speech tag sampled from the empirical tag distribution, then re-annotating the corpus; selectivity is the difference between original-task accuracy and control-task accuracy. For named-entity recognition, the baseline is the same probe trained on a randomly re-initialised copy of each pretrained model, giving a per-layer F1 gain; for news topic classification, the baseline is an untrained sequence model that supplies contextual but not linguistic information.","core_discovery":"The paper's central claim is that including a language in pretraining or multilingual adaptation changes the hidden representations themselves, not just the output layer: adapted models (AfroXLMR, Nguni-XLMR, AfriBERTa, AfroLM) show higher probe selectivity and higher probe gains than XLM-R on the same languages. The authors identify cross-lingual representation learning, especially among related Bantu languages, as the mechanism behind previously reported zero-shot performance on MasakhaNER. They further argue that this transfer is limited by language-family representation: Luganda benefits from Bantu-family transfer even when absent from pretraining, while Igbo, of the under-represented Volta-Niger family, shows the weakest encoding. The layer-wise picture—syntax peaking in middle-to-last layers and semantics spread across layers—matches earlier probing results in other languages and is presented as evidence that these models organise African-language information in familiar ways.","pith_inferences":["A direct test of AfroLM's high part-of-speech selectivity would be to train an identical model on randomly selected training samples rather than self-active samples; if selectivity drops, active learning is indeed selecting syntactically informative data.","The named-entity recognition random-reinitialisation baseline could be replaced or supplemented by a span-level control task that randomly reassigns entity types to multi-word spans, giving named-entity recognition the same interpretability as part-of-speech tagging.","Luganda's high news-topic variance suggests that cross-lingual semantic transfer is weaker than syntactic transfer; this predicts that sentence-level probing on distant language pairs will show larger drops than word-level probing at matched data sizes.","If selectivity curves are stable across languages, early-exit and layer-dropping strategies for African-language models should preserve most syntactic competence; this is testable by rerunning MasakhaPOS probes on truncated models."],"forward_implications":["Fine-tuning or probing middle-to-last layers should be the most sample-efficient route for syntactic tasks like part-of-speech tagging and named-entity recognition in African-language models.","Multilingual adaptation should be treated as a representational intervention: it improves downstream performance by enhancing language-specific knowledge inside hidden layers, not merely by adding a task head.","Zero-shot named-entity recognition should be expected when the target language shares a family with languages in the model's pretraining data, but not when the entire family is absent or under-represented.","Probe selectivity curves offer a practical diagnostic for choosing which layers to freeze, prune, or early-exit in African-language models."],"supporting_citations":[{"why":"Provides the MasakhaPOS dataset and the reported accuracies, including the lower Igbo results that the paper's selectivity patterns align with.","marker":"Dione et al., 2023"},{"why":"Supplies the MasakhaNER dataset and the previously reported zero-shot gains that the paper attributes to cross-lingual representation learning.","marker":"Adelani et al., 2022"},{"why":"Supplies the MasakhaNEWS dataset used for sentence-level topic classification probes.","marker":"Adelani et al., 2023"},{"why":"Defines the control-task method and the selectivity measure that carry the part-of-speech interpretation.","marker":"Hewitt and Liang, 2019"},{"why":"Provides AfroXLMR, the adapted model whose gains over XLM-R are the main evidence for the paper's central claim.","marker":"Alabi et al., 2022"},{"why":"Provides XLM-R, the massively multilingual baseline and the architecture that the adapted models build on.","marker":"Conneau et al., 2020"},{"why":"Provides AfriBERTa, the small from-scratch baseline that helps separate the effect of pretraining scale from adaptation.","marker":"Ogueji et al., 2021"},{"why":"Provides AfroLM, the self-active-learning model that shows the highest part-of-speech selectivity on most languages.","marker":"Dossou et al., 2022"},{"why":"Provides Nguni-XLMR, the focused Bantu adaptation that performs best for isiXhosa and Luganda.","marker":"Meyer et al., 2024"}],"fun_headline_variants":["Adapted African models store more linguistic knowledge in layers","Probing shows adaptation, not scale, encodes African grammar","Bantu family transfer keys African language probe gains","Unseen Luganda still benefits from Bantu-language pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons assume that probe scores, after subtracting control and random baselines, reflect genuine differences in what models know about a language rather than differences in how easily the probe can extract that knowledge from each model's representations.","fun_headline_variants_meta":{"raw":{"variants":["Adapted African models store more linguistic knowledge in layers","Probing shows adaptation, not scale, encodes African grammar","Bantu family transfer keys African language probe gains","Unseen Luganda still benefits from Bantu-language pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000527,"raw_usage":{"total_tokens":2516,"prompt_tokens":889,"completion_tokens":1627,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1562}},"tokens_in":505,"tokens_out":1627,"duration_ms":12915,"temperature":1.0,"reasoning_tokens":1562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:16:12.626413+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the word identities used to build hidden representations by shuffling the token embedding table before feeding text through the model, then retrain the probe; if the measured selectivity does not drop sharply, the probe is recovering surface word-type statistics rather than the linguistic knowledge the paper attributes to the model.","supporting_citations":[{"cited_title":"Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F","cited_arxiv_id":null,"evidence_quote":"Supplies the MasakhaNER dataset and the previously reported zero-shot gains that the paper attributes to cross-lingual representation learning."},{"cited_title":"Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F","cited_arxiv_id":null,"evidence_quote":"Provides AfroXLMR, the adapted model whose gains over XLM-R are the main evidence for the paper's central claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides AfriBERTa, the small from-scratch baseline that helps separate the effect of pretraining scale from adaptation."}],"review_version":1}