{"id":"23b8eb25-762f-48a3-a41b-6bb8222a25d6","arxiv_id":"2509.05668","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Llama-GENBA-10B is a 10B-parameter trilingual model that reports top Bavarian scores among sub-10B models on a machine-translated benchmark the authors built.","lead":"The authors trained a 10-billion-parameter language model on English, German, and Bavarian text, expanding Meta's Llama 3.1 with extra blocks and fine-tuning it for instructions. Their twist: Bavarian, a dialect with almost no training data, was added only in the final 10% of training, and the tuned model claims the best Bavarian benchmark scores among models its size.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Bavarian benchmark validity is the load-bearing risk: machine-translated, unreleased, and potentially generated by the same Gemini-flash model used for fine-tuning data.","rationale":"The reader's weakest assumption—that the Bavarian benchmark is a valid measuring instrument—is exactly the load-bearing point. The paper's strongest claim depends on a benchmark that is described only by a truncated sentence in §5.1, with no release, no human validation, and no translation protocol. My analysis adds a sharper mechanism: the same translator (Gemini-flash) used to create Bavarian fine-tuning data in §4.4 may also have created the benchmark, which would directly bias the comparison in favor of Llama-GENBA-10B-instruct. This is not an accusation; it is a possibility the paper does not rule out. The proposed test—an independent human translation of a sample of the benchmark—would settle whether the model's Bavarian advantage persists on authentic Bavarian. The reader's CONDITIONAL verdict remains appropriate: the concern is serious but fixable, and I do not see a load-bearing error that would falsify the core contribution outright. Therefore no verdict change is needed.","tokens_in":13503,"tokens_out":5606,"duration_ms":59541,"concrete_test":"Commission native Bavarian speakers to independently re-translate a stratified random sample of 100 items per benchmark (ARC-C, HellaSwag, TruthfulQA, MMLU, WinoGrande) from the German versions into Bavarian using a documented protocol, then run Llama-GENBA-10B-instruct, Apertus-8B-Instruct-2509, and gemma-2-9B-it on this independently translated set. If GENBA-10B-instruct does not remain at the top, or its margin over the baselines collapses, the §5.2 claim is an artifact of the original machine-translated evaluation instrument; if it does remain at the top, the concern is resolved. In the same pass, check the released benchmark against the 867k-pair fine-tuning corpus for n-gram overlap to rule out contamination.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5.2—that Llama-GENBA-10B-instruct outperforms all evaluated systems on Bavarian and is the best sub-10B model for the language—rests entirely on the Bavarian benchmark introduced in §5.1. That instrument is described in one incomplete sentence: 'For Bavarian we translated the German versions of these benchmarks to enable evaluation in' (the sentence is truncated). No translation system, prompt, human validation, back-translation check, or release of the test set is provided. This creates a concrete, load-bearing confound: §4.4 states that all Bavarian post-training data (Dolly, Alpaca, OpenHermes pairs) were produced by automatic translation into Bavarian using the Gemini-flash model. If the benchmark translations also used Gemini-flash—which the paper neither states nor rules out—then Llama-GENBA-10B-instruct was fine-tuned on Gemini-flash's Bavarian translationese, and the evaluation may be measuring fluency in that translationese rather than authentic Bavarian. Baselines such as Apertus-8B-Instruct-2509 and gemma-2-9B-it were not adapted on this same translationese, making the comparison biased. Additionally, an unreleased test set cannot be checked for overlap with the 867k-pair fine-tuning corpus. The headline result is therefore currently a claim about a private, machine-generated approximation of Bavarian, not a verified claim about the language itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Llama-GENBA-10B, a 10B-parameter trilingual model obtained by block-expansion continual pretraining of Llama-3.1-8B on 164B tokens (82B English, 82B German, 80M Bavarian). The development pipeline includes corpus construction, a trilingual tokenizer extension, a language-ratio experiment, staged introduction of Bavarian data, and supervised fine-tuning on 867k instruction pairs. The central claims are that the base model is competitive in English and German and strong in Bavarian, and that the instruction-tuned variant is the best sub-10B model on Bavarian, surpassing Apertus-8B-Instruct-2509 and gemma-2-9B-it. Evaluation is performed with English and German versions of ARC, HellaSwag, TruthfulQA, MMLU, and WinoGrande, plus Bavarian versions obtained by translating the German benchmarks. The paper also reports energy consumption for pretraining on a Cerebras CS-2.","tokens_in":13823,"tokens_out":4683,"duration_ms":52362,"significance":"If the Bavarian benchmark were validated and released, the paper would offer a useful, resource-efficient blueprint for adding a low-resource dialect to an existing LLM, and the documented energy measurement is a concrete contribution. The 1:1 English/German ratio experiments and the staged Bavarian data integration are informative. However, the headline result is not currently establishable: the Bavarian evaluation instrument is described in one truncated sentence, is not released, and is not validated against human judgments or existing Bavarian resources. The central SOTA claim is therefore conditional on an unverified benchmark.","major_comments":[{"comment":"The central claim that Llama-GENBA-10B-instruct is the best sub-10B Bavarian model rests entirely on the Bavarian benchmark introduced in §5.1. That section contains only the truncated sentence \"For Bavarian we translated the German versions of these benchmarks to enable evaluation in\" and provides no translation system, prompt, human validation, back-translation check, or release of the test set. This is load-bearing because §4.4 states that all Bavarian post-training data were produced by automatic translation with Gemini-flash; if the benchmark translations also use Gemini-flash, the fine-tuned model is evaluated on the same translationese it was trained on, which biases the comparison against baselines that were not adapted to that translationese. An unreleased test set also cannot be checked for overlap with the 867k-pair fine-tuning corpus. Until the benchmark is released and valid","section":"§5.1 (Bavarian benchmark) and §5.2 (headline claim)"},{"comment":"All comparisons are reported as single point estimates with no error bars, repeated runs, or significance tests. In Table 6, the ARC difference between Llama-GENBA-10B-base (0.3776) and EuroLLM-9B (0.3771) is 0.0005, yet the text describes the model as outperforming EuroLLM-9B. With a single evaluation run, such differences are within evaluation noise. Please provide confidence intervals, multiple seeds, or at least standard errors for the key comparisons, and report Bavarian results per benchmark rather than only through aggregate figures.","section":"§5.1, Table 6"},{"comment":"The 1:1 English/German language-mix ratio is selected using small 16B-token experiments reported in Table 3. The paper does not state which evaluation tasks are used for the Acc-EN/Acc-DE numbers. If these are the same ARC/HellaSwag-style tasks that later define success in §5, then the 1:1 choice is a model-selection step on the evaluation metric, making the later \"outperforms EuroLLM\" comparisons less clean. Please either specify that a separate held-out set was used, or acknowledge the selection effect and report the evaluation on a distinct, pre-registered test set.","section":"§4.1, Table 3"}],"minor_comments":[{"comment":"The sentence describing Bavarian benchmark construction is truncated: \"For Bavarian we translated the German versions of these benchmarks to enable evaluation in\". The missing text must be restored, and the translation methodology must be fully specified.","section":"§5.1"},{"comment":"Table 6 does not indicate which language the scores refer to. The text reports results separately for English, German, and Bavarian, but the table appears to present only a single set of numbers. Please provide separate tables or clearly labeled language columns.","section":"Table 6"},{"comment":"The bar charts are difficult to read and do not show numeric values or uncertainty. Add numeric labels and, where available, error bars so readers can verify the claims in the text.","section":"Figures 3 and 4"},{"comment":"The \"fertility score\" is used to select the tokenizer expansion but is never defined. Please provide the formula and also report downstream task performance for the candidate tokenizers, since fertility alone does not determine model quality.","section":"§3.2"},{"comment":"The conclusion states that the base model achieves \"state-of-the-art performance in Bavarian,\" while §5.1 reports that it ranks fourth among baselines in Bavarian. This inconsistency should be corrected.","section":"§6"},{"comment":"Some references are incomplete or appear mislabeled (e.g., the Eurostat 2024 citation is listed under Scientific Reports). Please verify all entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline result depends on a private, machine-translated benchmark that is neither released nor human-validated. I would ask the authors to release the Bavarian benchmark, document the translation protocol, and provide human validation before the claim of state-of-the-art Bavarian performance can be accepted. The English/German results are more solid but would also benefit from uncertainty quantification. If the benchmark cannot be released, the authors should downgrade the Bavarian claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Llama-GENBA-10B. It's a real artifact and a plausible recipe for low-resource dialect adaptation, but the headline claim — best sub-10B Bavarian model — currently rests on a machine-translated benchmark that isn't released and wasn't validated. That's the whole ballgame, and it's fixable.\n\nWhat's actually new: the model itself (base and instruct), the first Bavarian version of the standard five-task suite, and a documented energy cost of 35 MWh on a CS-2. The training recipe is a recombination of known techniques — block expansion, continual pretraining, language mixing — but the combination with a genuinely neglected dialect and the staged introduction of Bavarian data is a sensible contribution. The English and German numbers look like a competent 164B-token continual pretraining run; nothing embarrassing there. The tokenizer work with fertility scores is reasonable, and the energy reporting is genuinely useful for smaller labs.\n\nSoft spots. The Bavarian benchmark is described in a single truncated sentence and is not released. The fine-tuning data was translated into Bavarian with Gemini-flash, so if the benchmark translations used the same model — the paper neither says nor rules out — the evaluation may be measuring fluency in Gemini-flash's Bavarian translationese, which would bias the comparison against baselines that weren't adapted on that translationese. Also, no human validation, no back-translation check, no error bars. The headline SOTA claim is therefore not yet a claim about Bavarian as a language; it's a claim about a private, machine-generated approximation. That's a load-bearing flaw but not a fatal one: release the benchmark, add human checks, report variance, and the claim becomes testable.\n\nMinor quibbles: the 1:1 English/German ratio was chosen on the same ARC/HellaSwag suite used later in evaluation, so there's a mild selection-on-the-evaluation-instrument issue, though not circular in a damaging way since the final comparison is against external models. Point estimates differ by 0.005 and are described as outperforming; that's stats theater, but it doesn't affect the main story.\n\nWho is this for? People working on low-resource dialects and resource-constrained continual pretraining. It's a useful blueprint, and the energy data is a nice addition. I'd send it to peer review with a request for major revision: the Bavarian benchmark must be released and validated before the SOTA claim can stand. If that's done, the paper becomes a solid contribution.","headline":"A genuinely useful low-resource adaptation recipe whose headline Bavarian SOTA claim currently rests on an unvalidated, unreleased machine-translated benchmark — fixable, but not publishable as-is.","tokens_in":14389,"tokens_out":1788,"would_cite":false,"duration_ms":18220,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A continually pretrained 10B model, built from Llama-3.1-8B with only 80M Bavarian tokens, claims the top sub-10B Bavarian benchmark rank after instruction tuning.","keywords":["multilingual language model","continual pretraining","Bavarian dialect","low-resource language modeling","block expansion","trilingual evaluation","instruction tuning","English-German-Bavarian"],"falsifier":"Back-translate the Bavarian versions of ARC/HellaSwag/MMLU/TruthfulQA/WinoGrande into German and compare with the originals; if many items change their correct answer, leak the answer through translationese, or prove much easier for native speakers than the German originals, the claimed Bavarian superiority may be an artifact of the test rather than the model.","tokens_in":13379,"feed_emoji":"🗣️","tokens_out":10065,"duration_ms":101544,"temperature":0.7,"pith_summary":"The paper sets out to show that a foundation model can serve English, German, and a low-resource dialect fairly without letting English dominate the training mix, and that even a very small dialect corpus can produce a state-of-the-art dialect model when it is introduced late in training. Starting from Llama-3.1-8B, the authors insert eight new Transformer blocks, continue training on 164B tokens (82B English, 82B German, 80M Bavarian), and then instruction-tune on a trilingual set of 867k prompts. Their central claim is that the fine-tuned model is the best sub-10B system on Bavarian benchmarks, ahead of gemma-2-9B-it and Apertus-8B-Instruct-2509, while matching EuroLLM-9B in German and beating it in English. If the claim holds, it matters because it offers a comparatively cheap recipe for adding an under-resourced dialect to an existing model, together with the first trilingual English-German-Bavarian benchmark suite. The evidence is accuracy on translated versions of ARC, HellaSwag, MMLU, TruthfulQA, and WinoGrande.","feed_headline":"10B model tops Bavarian tests with just 80M dialect tokens","feed_subtitle":"Expanded from Llama-3.1-8B and fine-tuned on translated instruction data, it outranks gemma-2-9B and Apertus-8B in Bavarian.","key_machinery":"The mechanism is a staged continual-pretraining recipe. Eight Transformer blocks are inserted into a frozen Llama-3.1-8B backbone, with the new attention-output and FFN-final layers zero-initialized, so the expanded 10B model preserves the base's knowledge while adding trainable capacity. For the first 90% of iterations the model sees only a 1:1 English-German mix, chosen after a 16B-token experiment showed that ratio beat a 9:1 German-heavy split; in the final 10%, Bavarian text is introduced, upsampled to 80M tokens. A tokenizer expanded by 20% with German and Bavarian subword units lowers token fertility. The paper credits this sequence—reserve new capacity, establish the high-resource la","core_discovery":"The discovery the paper argues for is that Bavarian competence does not require a large Bavarian corpus. With Bavarian capped at 80M tokens—about 0.05% of the 164B-token continued-pretraining budget—the base model already lands near gemma-2-9b, Apertus-8B-2509, and Llama-3.1-8B on Bavarian tasks, and the instruction-tuned variant takes first place among all evaluated sub-10B models, outperforming gemma-2-9B-it and Apertus-8B-Instruct-2509. The paper also claims the same model beats EuroLLM-9B on English and matches it on German, suggesting the balanced 1:1 English-German mix did not cost English strength. These results are presented as evidence that block-expansion continual pretraining, wit","pith_inferences":["Editorial inference: the staging result suggests the timing of low-resource data exposure may matter more than its raw volume; an ablation that feeds Bavarian throughout training, or doubles the late-stage Bavarian budget, would test that directly.","Editorial inference: the same 'translate a high-resource benchmark, then reuse it' template could be applied to other German dialects, but each new test set would need its own native-speaker validation before its leaderboard is trusted.","Editorial inference: the energy and wall-clock numbers are specific to the CS-2 hardware, so the 'efficient for small teams' message should be read as a hardware-dependent data point rather than a universal cost guarantee."],"forward_implications":["A dialect can become a model's best language with a token share below 0.1%, if the dialect data arrive after the high-resource backbone is stable.","A 1:1 English-German continued-pretraining mix supports strong English and German results simultaneously, so English dominance is not required for a capable European model.","Instruction tuning on machine-translated English/German instruction data is enough to convert a competitive base model into the leading sub-10B Bavarian system.","The 35.23 MWh / 66-day run on a single Cerebras CS-2 offers a concrete energy baseline for small teams attempting similar multilingual continual pretraining.","The translated ARC/HellaSwag/MMLU/TruthfulQA/WinoGrande suite becomes a reusable trilingual benchmark for future German-English-Bavarian systems."],"supporting_citations":[{"why":"Supplies the Llama-3.1-8B checkpoint that GENBA-10B expands and continually pretrains, defining the model family and base capacity.","marker":"Dubey et al., 2024"},{"why":"Contributes the block-expansion method that grows 8B to 10B with zero-initialized inserted blocks.","marker":"Wu et al., 2024"},{"why":"Extends block expansion to bilingual adaptation, the direct precursor for using it in a trilingual setting.","marker":"Gosal et al., 2024"},{"why":"Supplies GlotLID-m, the classifier used to extract 262k Bavarian sentences from FineWeb and build the scarce Bavarian partition.","marker":"Kargaran et al., 2023"},{"why":"Provides the Arabic-English mixing evidence that motivates testing a 1:1 German-English ratio, which the paper then selects.","marker":"Sengupta et al., 2023"},{"why":"Supplies the Gemini-flash model that the paper uses for the Bavarian instruction-data translations; the same translation route is the only stated mechanism for producing Bavarian evaluation items.","marker":"Comanici et al., 2025"},{"why":"Supplies the German-adapted evaluation harness used to score German and Bavarian tasks.","marker":"Dac Lai et al., 2023"},{"why":"Supplies the evaluation harness that runs the five benchmarks and produces the model comparisons.","marker":"Gao et al., 2024"},{"why":"Defines the EuroLLM-9B baseline that GENBA-10B is claimed to beat in English and match in German.","marker":"Martins et al., 2025"}],"fun_headline_variants":["10B model tops Bavarian with 80M tokens","Bavarian leader despite tiny 80M corpus","80M Bavarian tokens, top model results","Bavarian crown with just 80M tokens","Small dialect data, big model win"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline Bavarian ranking stands on the assumption that the machine-translated Bavarian benchmark items measure Bavarian ability with the same difficulty and validity as the German originals, and the paper reports no human validation of those translations.","fun_headline_variants_meta":{"raw":{"variants":["10B model tops Bavarian with 80M tokens","Bavarian leader despite tiny 80M corpus","80M Bavarian tokens, top model results","Bavarian crown with just 80M tokens","Small dialect data, big model win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1571,"prompt_tokens":861,"completion_tokens":710,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":605,"completion_tokens_details":{"reasoning_tokens":637}},"tokens_in":605,"tokens_out":710,"duration_ms":8107,"temperature":1.0,"reasoning_tokens":637,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:14:02.302870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Back-translate the Bavarian versions of ARC/HellaSwag/MMLU/TruthfulQA/WinoGrande into German and compare with the originals; if many items change their correct answer, leak the answer through translationese, or prove much easier for native speakers than the German originals, the claimed Bavarian superiority may be an artifact of the test rather than the model.","supporting_citations":[{"cited_title":"Bilingual Adaptation of Monolingual Foundation Models","cited_arxiv_id":"2407.12869","evidence_quote":"Extends block expansion to bilingual adaptation, the direct precursor for using it in a trilingual setting."},{"cited_title":"GlotLID: Language Identification for Low-Resource Languages","cited_arxiv_id":"2310.16248","evidence_quote":"Supplies GlotLID-m, the classifier used to extract 262k Bavarian sentences from FineWeb and build the scarce Bavarian partition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the German-adapted evaluation harness used to score German and Bavarian tasks."}],"review_version":1}