{"id":"0e828a7f-702c-4554-a5d4-0a9778715adc","arxiv_id":"2412.12956","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SnakModel, an open Danish 7B LLM, outperforms other Llama2-7B-based models on the ScandEval Danish benchmark, with analyses of training dynamics and data curation.","lead":"This paper trains and releases SnakModel, a Danish language model built by further training Llama2-7B on 13.6 billion Danish words and 3.7 million Danish instructions. It reports that SnakModel beats other Llama2-7B-based models on eight Danish benchmarks, and it shares practical lessons about data curation and training choices for smaller languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Leakage audit in §4.1 puts DANSK NER and ScaLA evaluation data inside SnakModel's pretraining corpus (plus 6/200 AngryTweets), so Table 3's 56.63 average is not clean evidence; the headline Llama2-7B comparison needs an overlap-quantified, clean-subset re-evaluation before it can be taken at face…","rationale":"The paper's main empirical claim is a benchmark comparison, and the benchmark is the only quantitative evidence for it. Section 4.1 openly discloses contamination, which is a validity threat to the reported per-task scores and to the training-dynamics conclusions for LA, NER, and SENTI. The authors deserve credit for disclosing this and for releasing scripts, but disclosure is not mitigation: no overlap fraction or clean-subset score is provided. The tasks with source-corpus overlap are exactly where SNAKMODEL's biggest or most fragile advantages over LLAMA2-7Bchat+INSTda appear, so the reported 56.63 average cannot be taken at face value as evidence of generalization. The central comparison may still hold because TM and CT show large, apparently uncontaminated gains, but the paper does not present the clean computation. This supports the reader's CONDITIONAL verdict; I would not escalate to REJECT because the disclosure, open artifacts, and non-overlapping task margins make it plausible the headline claim survives. The required next step is an overlap-quantified, clean-subset re-evaluation. There is no formal verification, but the open data and evaluation scripts are real independent support and make the proposed test straightforward to run.","tokens_in":18344,"tokens_out":8745,"duration_ms":86427,"concrete_test":"Using the released corpus scripts and the text-dedup pipeline, build the exact deduplicated SNAKMODEL pre-training set. For the official ScandEval test splits of DANSK (NER), ScaLA (LA), and AngryTweets (SENTI), compute exact 8-gram and MinHash near-duplicate overlap rates. Then re-run the Table 3 evaluation of SNAKMODEL-7Binstruct and LLAMA2-7Bchat+INSTda on the non-overlapping subsets and reconstruct the average. If the clean average remains clearly above the baseline (say at least 3 points, retaining the 5.24 headline gap), the concern is resolved; if it falls to near or below the baseline, the headline claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SNAKMODEL-7Binstruct outperforms all other LLAMA2-7B-based models with the same Danish instruction data is supported only by the ScandEval test scores in Table 3. The paper's own leakage check (Section 4.1) says the DANSK NER dataset was 'completely included (without labels)' because it was sampled from Gigaword, that 'many parts' of the ScaLA dataset were included in original form in Gigaword and CC100, and that 6/200 sampled AngryTweets tweets were found in the Twitter data. The paper never reports how much of the actual ScandEval test splits overlap the deduplicated pre-training corpus, and it never recomputes scores on clean subsets. These are the exact tasks where SNAKMODEL's margin over the strongest +INSTda baseline is either large or fragile: LA +9.51, SENTI +0.78, NER +0.06. If overlap inflates those scores, the average 56.63 and the internal training-dynamics story for LA/NER are not trustworthy. The comparison may still survive because TM and CT strongly favor SNAKMODEL, but the paper does not quantify that, so the central claim currently rests on an unmeasured contamination level.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SnakModel, a Danish LLM created by continuously pre-training Llama2-7B on 13.6B Danish words and then instruction-tuning on 3.7M Danish instructions. It evaluates the model on the Danish portion of the ScandEval benchmark across eight tasks, compares it with contemporary Llama2-7B- and Mistral-7B-based models, analyzes intermediate training dynamics and weight divergence, and distills recommendations for adapting LLMs to lower-resource languages. The central claim is that SnakModel-7Binstruct outperforms all other Llama2-7B-based models, including those trained on the same Danish instruction data, with an average score of 56.63 (Table 3).","tokens_in":18597,"tokens_out":5060,"duration_ms":46951,"significance":"The paper is a genuinely useful resource paper: it releases model weights, intermediate checkpoints, data-collection scripts, and evaluation code, and the +INSTda ablation is a well-designed control that isolates the effect of Danish pre-training from instruction tuning. The training-dynamics and weight-divergence analyses are informative and go beyond a single leaderboard. If the evaluation is confirmed to be clean, the guidance for mid-resource languages (e.g., one epoch of instruction tuning, focus on embedding/feed-forward updates) would be a valuable contribution.","major_comments":[{"comment":"The paper acknowledges in Section 4.1 that the DANSK NER dataset was \"completely included (without labels)\" in the pre-training corpus and that \"many parts\" of the ScaLA dataset were included in original form, with 6/200 AngryTweets tweets also found. Because the ScandEval test split used in Table 3 is built from these datasets, the reported NER and LA scores, and to a lesser degree SENTI, may reflect memorization of input text rather than generalization. The paper never quantifies the fraction of the actual test splits present in the deduplicated pre-training corpus, and it never recomputes the benchmark averages on overlap-free subsets. Since the largest margin over the strongest +INSTda baseline is on LA (+9.51) and the NER margin is only +0.06, the headline claim that SNAKMODEL outperforms all other Llama2-7B-based models is not yet supported on the contaminated tasks. Please report per-task overlap counts (exact and near-duplicate) and re-run Table 3 on clean subsets, or otherwise show that the 56.63 average and the rank ordering are robust to excluding contaminated instances.","section":"Section 4.1 (Leakage) and Section 5, Table 3"},{"comment":"All benchmark scores are reported for a single run, with no variance estimates or significance tests. Several task-level margins over the second-best Llama2-7B model are below one percentage point (NER +0.06, QA +0.26, SENTI +0.78), so the claim of \"outperforms all other LLAMA 2-7B-based models\" rests on differences that could easily be within run-to-run noise. I ask for at least bootstrap confidence intervals on the average, and ideally repeated instruction-tuning runs for the main comparison, to make the central claim statistically grounded.","section":"Section 5, Table 3"},{"comment":"The training-dynamics conclusion that \"instruction tuning after 2,000–5,000 steps ... may already be sufficient to obtain close-to-final performance\" is drawn from validation scores on the same contaminated tasks (LA and NER). If the LA and NER gains are partly due to the model having seen the test text during pre-training, the plateau pattern in Figure 2b may be an artifact of memorization rather than a genuine signal about when Danish pre-training becomes useful. Please verify the plateau on clean validation subsets before presenting this as guidance for future work.","section":"Section 4.3, Figure 2b, and Section 6"}],"minor_comments":[{"comment":"The leakage test samples \"200 random 8-grams from each of our datasets\" but does not specify whether the search was run against the pre-training corpus after deduplication or before it; please clarify, and also report exact whole-sequence overlaps with the test splits rather than only 8-gram hits.","section":"Section 4.1"},{"comment":"The statement that \"many parts\" of ScaLA were included in original form would be much more informative with a quantitative count or a proportion; as written, the reader cannot gauge the severity of the overlap.","section":"Section 4.1"},{"comment":"The \"AVG.\" column appears to be an unweighted average of eight task scores with different metrics; please state this explicitly and consider adding a note on how the average behaves if the contaminated tasks are removed.","section":"Table 3"},{"comment":"LoRA rank 128 is described as applied to \"all parameters within the model,\" but it is not stated whether this includes the input embeddings and the language-modeling head; this matters for the weight-divergence interpretation in Figure 4.","section":"Section 4.2"},{"comment":"The captions use \"SNAK MODEL\" with a space; please use \"SNAKMODEL\" consistently to match the rest of the paper.","section":"Figures 1 and 2"},{"comment":"The dashed line in Figure 2 is not explained in the caption; please identify what it represents.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid resources contribution for the Danish NLP community, and the authors should be commended for releasing code, data scripts, and intermediate checkpoints, as well as for reporting their leakage audit honestly. The unresolved issue is not a lack of disclosure but the absence of any quantification of the disclosed overlap and the lack of a clean-subset evaluation. Because the central comparison in Table 3 is the paper's main empirical claim, I would ask the editor to require the overlap-quantified re-analysis as a condition for acceptance rather than treating it as optional future work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, transparent engineering paper with real resources, but the headline average is not clean because the pretraining corpus overlaps two of the eight benchmark tasks. The authors disclose this, but they don't quantify how much of the test splits leaked, and they don't recompute scores on clean subsets. That leaves the exact margins over the +INSTda baseline uncertain on LA and NER.\n\nWhat's genuinely new: the 13.6B-word curated Danish corpus (released with scripts), the SnakModel-7B base/instruct checkpoints, intermediate checkpoints, and the training-dynamics analysis. The ablation across instruction templates and the weight-divergence analysis are useful and clearly reported. The +INSTda baseline is a good experimental choice: it isolates the effect of Danish continued pretraining against a strong control. The honesty about the leakage check and the compute details is refreshing.\n\nSoft spots, in order of importance. First, as above: DANSK NER is fully in the corpus (without labels) and large parts of ScaLA are too, so the LA (+9.51 over the best +INSTda baseline) and NER (+0.06) margins are suspect. The paper should either report overlap between the deduplicated pretraining corpus and the exact test splits, or re-run those tasks on clean subsets. Without that, the per-task story for training dynamics on LA and NER is not reliable. Second, there are no error bars or significance tests; a single run is common in this literature but makes the small margins (NER +0.06, SENTI +0.78) fragile. Third, the abstract's \"highest overall performance\" is slightly overbroad: Munin-v0.1-dev0 scores 61.28, above SnakModel's 56.63, and the paper's explanation rests on base-model strength. That's fair but should be stated more carefully.\n\nNone of this sinks the central claim. On TM, CT, and QA the margins are large and those tasks are native Danish, so the main qualitative finding — continued pretraining on native text helps culturally specific tasks — holds. The paper is exactly the kind of resource contribution that helps mid-resource language work.\n\nRecommendation: send it to peer review. Ask for a leakage quantification or clean-subset re-evaluation, and a small amount of hedging on the abstract. Deserves referee time.","headline":"Solid open Danish LLM and corpus with honest leakage disclosure, but the headline average rests on unmeasured overlap with two benchmark tasks; needs a clean-subset re-evaluation.","tokens_in":19194,"tokens_out":2454,"would_cite":true,"duration_ms":22895,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By continuously pre-training a 7-billion-parameter English model on 13.6 billion curated Danish words and tuning it on 3.7 million Danish instructions, this paper produces a Danish LLM that beats every other Llama-2-7B-based model on the…","keywords":["Danish language model","continuous pre-training","instruction tuning","ScandEval","low-resource NLP","training dynamics","weight divergence"],"falsifier":"Compare the test splits of DANSK NER and ScaLA against the SnakModel pre-training corpus: if a substantial fraction of test examples appears verbatim in the training data, re-evaluate on strictly disjoint versions; the central claim of superior generalization would collapse if the NER and LA gains disappear on those clean splits.","tokens_in":18086,"feed_emoji":"🇩🇰","tokens_out":7447,"duration_ms":60521,"temperature":0.7,"pith_summary":"The paper sets out to show that a strong Danish large language model can be built by taking an existing English 7B model, continuing its training on a heavily curated, deduplicated corpus of 13.6B Danish words, and then instruction-tuning on 3.7M Danish instruction–answer pairs. It reports that the resulting model, SnakModel-7Binstruct, reaches an average score of 56.63 on the Danish tasks of ScandEval, beating every other Llama-2-7B-based system, including ones given the same Danish instruction data. The authors also claim that most downstream gains appear within the first 2,000–5,000 continued-pretraining steps, that one epoch of instruction tuning is enough, and that parameter change concentrates in the embedding layer, the SwiGLU up-projection, and the language-model head. A sympathetic reader would care because the paper turns these observations into concrete training guidelines for small language communities with limited compute.","feed_headline":"Danish 7B model tops all Llama-2 rivals on ScandEval","feed_subtitle":"Curated 13.6B-word corpus and 3.7M Danish instructions lift average score to 56.63.","key_machinery":"The central mechanism is continued pre-training on a curated Danish corpus, evaluated through two diagnostic lenses: intermediate checkpoints scored on ScandEval to trace when Danish competence emerges, and principal subspace angles (a measure of parameter change) computed before and after adaptation to locate where it happens. The continuation trains Llama 2-7B for 12,500 steps on 13.6B words with a low peak learning rate of $1.5 \\times 10^{-5}$ to avoid gradient explosions, followed by one epoch of LoRA instruction tuning with rank 128. The subspace-angle analysis shows that most change concentrates in the embedding, the SwiGLU gate and up-projection, and the language-model head, with self-attention barely moving.","core_discovery":"The central claim is that continuous pre-training of Llama 2-7B on a large native Danish corpus, followed by instruction tuning, yields the best Llama-2-7B-based model for Danish, with an average ScandEval score of 56.63 versus 46.48 for the English base and 49.50 for the chat version. The largest gains appear on tasks built from natural Danish text rather than translations: linguistic acceptability rises from 33.43 to 52.91, proverb understanding from 38.69 to 71.05, and citizenship-test accuracy from 57.05 to 71.88. The paper also shows that continued pre-training temporarily degrades named-entity recognition and question answering, but that instruction tuning recovers both, and that close-to-final performance is reached after only a fraction of the pre-training corpus has been seen.","pith_inferences":["If the documented leakage is substantial, the reported NER and LA improvements may overstate true generalization; a clean held-out evaluation would put a realistic bound on the adaptation gain.","The parameter-localization result suggests that for typologically close languages, parameter-efficient targeting of embeddings and feed-forward layers could approximate full continued pre-training at a fraction of the cost—an extension the paper does not directly test.","The 2,000–5,000-step saturation point raises the question of whether even smaller corpora (a few billion words) would suffice for Danish, which could be tested by pre-training on progressively smaller slices.","The recipe may transfer to other Germanic mid-resource languages, but for typologically distant targets self-attention may need more updating, so the weight-divergence pattern should be re-measured per language family."],"forward_implications":["Danish language modeling can start from an existing English 7B model plus a ~13.6B-word curated corpus instead of training from scratch.","Instruction tuning after 2,000–5,000 steps of continued pre-training may reach near-final performance, so expensive long pre-training can be truncated.","A single epoch of instruction tuning on translated instruction data is enough to restore and amplify instruction-following behavior in the target language.","Because adaptation concentrates in the embedding, SwiGLU up-projection, and language-model head, future Danish-adaptation runs could train only those parameters for similar efficiency gains.","Native-language data, rather than translated data, is what drives gains on culture-specific tasks such as proverb understanding and citizenship questions."],"supporting_citations":[{"why":"Provides the Llama 2-7B base model and tokenizer that SnakModel is continuously pre-trained from; the starting point of all comparisons.","marker":"Touvron et al., 2023"},{"why":"Defines the ScandEval benchmark with the eight Danish tasks, splits, and the average score used to rank all models.","marker":"Nielsen, 2023"},{"why":"Supplies the Danish Gigaword corpus, a major pre-training source and the origin of the DANSK NER leakage concern.","marker":"Strømberg-Derczynski et al., 2021"},{"why":"Introduces the DANSK NER dataset, a benchmark task whose data was included (without labels) in the pre-training corpus.","marker":"Hvingelby et al., 2020"},{"why":"Provides the Aya Collection, the source of 3.6M of the 3.7M Danish instruction-answer pairs used for instruction tuning.","marker":"Singh et al., 2024"},{"why":"Supplies the principal-subspace-angle method used to measure parameter divergence and locate where Danish adaptation occurs.","marker":"Müller-Eberstein et al., 2024"}],"fun_headline_variants":["Danish LLM SnakModel beats Llama-2 rivals on ScandEval","Open Danish model tops Llama-2 family with curated data","SnakModel: 13.6B Danish words lift 7B Llama to top scores","How a 7B Danish model outranks all Llama-2 variants","SnakModel: Lessons in training LLMs for smaller languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation is only trustworthy as a measure of generalization if the documented overlap between the training corpus and the ScandEval data—the DANSK NER set was included verbatim and parts of ScaLA were included—does not inflate the reported NER and LA scores through memorization.","fun_headline_variants_meta":{"raw":{"variants":["Danish LLM SnakModel beats Llama-2 rivals on ScandEval","Open Danish model tops Llama-2 family with curated data","SnakModel: 13.6B Danish words lift 7B Llama to top scores","How a 7B Danish model outranks all Llama-2 variants","SnakModel: Lessons in training LLMs for smaller languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1372,"prompt_tokens":919,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":351}},"tokens_in":535,"tokens_out":453,"duration_ms":4199,"temperature":1.0,"reasoning_tokens":351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:32:45.743636+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the test splits of DANSK NER and ScaLA against the SnakModel pre-training corpus: if a substantial fraction of test examples appears verbatim in the training data, re-evaluate on strictly disjoint versions; the central claim of superior generalization would collapse if the NER and LA gains disappear on those clean splits.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ScandEval benchmark with the eight Danish tasks, splits, and the average score used to rank all models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Danish Gigaword corpus, a major pre-training source and the origin of the DANSK NER leakage concern."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the DANSK NER dataset, a benchmark task whose data was included (without labels) in the pre-training corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the principal-subspace-angle method used to measure parameter divergence and locate where Danish adaptation occurs."}],"review_version":1}