REVIEW 3 major objections 4 minor 3 references
DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper claims that DaLA, a new Danish benchmark built from fourteen real-world error types, is harder and more discriminative than the existing standard, with nine of ten tested models scoring 6.04% lower on average.
desk verdict A useful Danish acceptability dataset with honest validation, but the 'harder than ScaLA' claim needs a noise-aware comparison before it fully lands. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the set of fourteen corruption functions derived from an expert-curated list of common Danish problems (e.g., confusion of 'nogle'/'nogen', 'ligge'/'lægge', genitive forms, the '-ende'/'-ene' suffix, and the so-called 'r problem' for nouns, verbs and adjectives). Each function injects a single error into an acceptable sentence; an iterative algorithm applies rarest corruptions first to balance error-type representation across the dataset. A semi-automatic validation pipeline — automatic grammar detection followed by manual review by a native-speaker lexicographer — estimates the precision of each corruption type (average 95.7%), and Equation 1 re-estimates precisio
What would settle it
Take the two corruption types whose manual validation was extrapolated from subsets (pronoun corruption: 140 of 877 examples; suffix determiner: 164 of 1199) and annotate all remaining examples. If the re-estimated precision drops substantially below the reported 0.957 average, or if model comparisons on the high-precision subset (say, corruption types with precision ≥ 0.99) no longer reproduce the 6.04% average drop, then the claimed difficulty gap is partly label noise rather than task difficulty.
Extended reading notes
Core claim
DaLA converts common Danish writing errors into 14 corruption functions, applies them to sentences from a dependency treebank to create minimal pairs (one acceptable, one corrupted), validates each corruption's precision through an automatic grammar checker and manual expert review, and then tests ten open-weight LLMs. Nine out of ten models perform worse on DaLA than on the previous Danish benchmark, with an average drop of 6.04% in Matthews correlation; reasoning-oriented models fall most sharply (14.32% on average). The authors interpret the drop as evidence that DaLA is more representative of real-world Danish errors and has higher discriminatory power — it distinguishes a Danish-only la
Load-bearing premise
That every corrupted sentence is genuinely unacceptable: the paper's own validation measures average corruption precision at 95.7%, but two error types — genitive (80.0%) and han/hun-vs-det (82.4%) — leave roughly one in five corruptions possibly still acceptable, and failure is not necessarily spread evenly across models.
Editorial extensions
If this is right
- Danish LLM evaluation can now use a benchmark whose errors match what Danish speakers actually write, making the acceptability judgement task more realistic than the previous word-removal/word-swap baseline.
- The larger performance gaps on DaLA (e.g., reasoning models dropping by ~14.3 percentage points) mean the benchmark can separate high-performing from underperforming models more clearly.
- The corruption functions are reusable: because they require no annotation beyond a sentence being Danish and acceptable, DaLA can be extended to any new Danish sentence source or grown into larger versions (up to 7,656 sentences).
- Reporting scores per corruption type enables error-specific diagnostics, locating exactly which Danish constructions a model handles poorly.
- The methodology — derive corruption functions from a language's common attested errors — transfers to other low-resource languages for which such error lists exist.
Reading between the lines
- If the difficulty gap is driven by representativeness rather than label noise, then DaLA's model ranking likely predicts real-world Danish writing support quality better than ScaLA; a practical test would be whether the ordering holds on naturally occurring learner texts.
- The two lowest-precision corruption families (genitive at 80%, han/hun-vs-det at 82.4%) are precisely the ones where an ungrammatical form can coincide with a dialectal or informal variant; re-annotation of the full sets would show whether those error types should be tightened or excluded from hard thresholds.
- The sharp drop for reasoning models invites a concrete follow-up: probe whether these models fail because DaLA's errors are surface-form confusions ('nogle'/'nogen') that expose lexical rather than syntactic generalization, which would point to where training data for Danish is thin.
- Since the corruption functions are deterministic and language-specific, a cross-language version of this pipeline could produce comparable hardness metrics across Nordic languages, where shared error patterns (like the r-problem) are common.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DaLA, a Danish linguistic acceptability benchmark constructed by applying 14 corruption functions, derived from an inventory of common real-world Danish errors, to sentences from the Universal Dependencies Danish corpus. The authors describe a hybrid automatic/manual validation procedure, report average corruption precision of 95.7%, and evaluate ten open-weights LLMs under the EuroEval framework. Comparing DaLA with the existing ScaLA benchmark, they report that most models score lower MCC on DaLA, with an average drop of 6.04% and larger drops for reasoning models, and they interpret this as evidence that DaLA is harder and has higher discriminatory power. They also release code and datasets at three sizes.
Significance. If the central claims hold, DaLA is a useful resource for Danish NLP: it is grounded in a documented set of frequent Danish errors, it is not tied to a particular annotation format, and it provides a public, reproducible evaluation protocol with open-weight models. The paper's strengths include the use of an external error inventory, the combination of automatic and manual corruption checking, and the release of code and data. However, the difficulty and discriminatory-power claims rest on the validity of the negative labels and on a consistent evaluation setup; as detailed below, both aspects need additional support before the central conclusions can be accepted.
major comments (3)
- [§1 vs §3.5] The manuscript gives contradictory dataset splits. §1 states 'the DaLA dataset consists of 1,664 sentence pairs, with 512 for training, 1,024 for validation, and 128 for testing' (3,328 samples). §3.5 reverses the validation and test sizes: 'Training: 512 pairs', 'Validation: 128 pairs', 'Test: 1024 pairs'. Since the stated goal is comparability with ScaLA and Table 2 quantifies the difficulty gap, the actual split used for evaluation must be stated unambiguously. If §3.5 is correct, §1 is wrong in both the split sizes and the claimed comparability; if §1 is correct, §3.5 and the relationship to the '7,656 sentences' description need a clear explanation.
- [§4.3, Table 1, §5.2] The central 'harder than ScaLA' claim is vulnerable to label noise from corruptions that remain acceptable. The average precision is 95.7%, and two types are much lower (genitive 0.80, han/hun 0.824). With balanced classes, an ε noise rate in the corrupted class caps a perfect model's MCC at sqrt((1−ε)/(1+ε)); at ε=0.043 this is ≈0.96, already 4 points below perfect. The observed average drop is 6.04 points, so noise could account for a large part of the claimed difficulty difference, and uneven noise across corruption types could inflate the 'discriminatory power' result. The precision estimates themselves are extrapolated from 140/877 and 164/1199 examples via Eq. (1) with no confidence intervals; 'recurrent linguistic error patterns' is not a statistical guarantee. Spelling errors are excluded from validation yet appear in the benchmark. Please report per-type precision with intervals
- [§5.2] The secondary claim of 'higher discriminatory power' is not established by the reported evidence. Table 2 shows confidence intervals that overlap for most models; the only visible separation is the two reasoning models, and the 'average decline of 14.32%' for that group is based on two models. No statistical test is provided (e.g., comparison of score variance, or a model-type × dataset interaction). The conclusion is inferred post hoc from the same numbers used to compute difficulty. This claim should be either toned down or supported by an appropriate analysis.
minor comments (4)
- [Abstract vs §5.2] The abstract reports a maximum drop of '14.55%', while §5.2 reports a range 'from 1.71% to 14.85%'. Harmonize the numbers.
- [§4.3, Table 1] Spelling errors are said to be excluded from validation, but they are part of the evaluated benchmark. Clarify whether the 0.957 average precision includes this corruption type and how its precision was estimated.
- [Eq. (1)] The notation in Eq. (1) is compact and used before all terms are defined. Please state explicitly that tpauto/fpauto are the automatic-validation counts and tpman/fpman are the manual-annotation counts from the subset sampled from fpauto.
- [Title page] Author name formatting: 'Peter Schneider-Kamp,Lukas Galke Poech' appears without a space after the comma; this should be corrected.
Circularity Check
No significant circularity: DaLA construction, validation, and LLM evaluation are independent; the difficulty claim is an empirical measurement, not a construction.
full rationale
The central claim that DaLA is harder than ScaLA rests on observed LLM MCC scores on held-out test sets (Table 2), not on any parameter fitted to those scores. The corruption functions are derived from external linguistic resources (Typiske Problemer; the Danish Dictionary spelling list) and applied to UDD sentences; this is independent of the evaluation outcome. The precision validation in Section 4 is a quality check, not a prediction: Eq. (1) is a standard correction for imperfect automatic screening, and the paper openly states that the manual subsets are 140/877 and 164/1199 and that the estimate is argued from recurrent patterns, not derived from the benchmark's own target claim. The comparison with ScaLA uses the same splits and the same EuroEval framework, and the inclusion of the two 'basic corruptions' from Nielsen (2023) is a controlled component, not a self-referential premise. The only potentially self-authored citations (Sørensen et al. 2023; Appel et al. 2024) supply external lexicographic corpus data for spelling errors; even with author overlap, this is data input, not a load-bearing uniqueness or circular derivation. No uniqueness theorem, ansatz-by-citation, or renaming pattern is present. Concerns about label noise in low-precision corruption types (genitive 0.800, han/hun vs det 0.824) and about the post hoc adjustment of evaluation parameters are legitimate correctness risks, but they are not circularity: the benchmark's construction does not reduce to its evaluation outcome.
Assumptions & free parameters
free parameters (3)
- Error-selection ranking scores =
not disclosed (0–5 scale)
- Per-corruption conservativeness filters =
not disclosed
- Evaluation hyperparameters (patience, runs) =
patience=20, runs=50
assumptions (6)
- domain assumption Sentences from the UD Danish corpus are acceptable.
- domain assumption The Typiske Problemer list by the Danish Language Council represents the most common real-world Danish errors.
- domain assumption writeassistant's grammaticality judgments are sufficiently accurate.
- domain assumption Single-error minimal-pair corruption captures linguistic acceptability.
- domain assumption Manual annotator's decisions are ground truth for corruptedness.
- ad hoc to paper Extrapolation from annotated subsets (Eq. 1) is valid for the two largest corruption types.
Cite this review
Pith. "Pith review of DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors." pith.science (2026). https://pith.science/paper/CPYPVCZJ
@misc{pith2026251204799,
author = {Pith},
title = {Pith review of: DaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors},
year = {2026},
howpublished = {\url{https://pith.science/paper/CPYPVCZJ}},
note = {Machine review of arXiv:2512.04799}
}
read the original abstract
We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we introduce a set of fourteen corruption functions that generate incorrect sentences by systematically introducing errors into existing correct Danish sentences. To ensure the accuracy of these corruptions, we assess their validity using both manual and automatic methods. The results are then used as a benchmark for evaluating Large Language Models on a linguistic acceptability judgement task. Our findings demonstrate that this extension is both broader and more comprehensive than the current state of the art. By incorporating a greater variety of corruption types, our benchmark provides a more rigorous assessment of linguistic acceptability, increasing task difficulty, as evidenced by the lower performance of LLMs on our benchmark compared to existing ones. Our results also suggest that our benchmark has a higher discriminatory power which allows to better distinguish well-performing models from low-performing ones.
Figures
Reference graph
Works this paper leans on
-
[3]
silent d mistakes
Judging grammaticality: Experiments in sen- tence classification.Wagner, Joachim and Foster, Jennifer and van Genabith, Josef (2009) Judging grammaticality: experiments in sentence classifica- tion. CALICO Journal, 26 (3). pp. 474-490. ISSN 0742-7778, 26. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey, Wei Peng, Sheng-Fu Wang, and Samuel R. ...
2009
-
[2013]
Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L Frank
A comparison of informal and formal accept- ability judgments using a random sample from lin- guistic inquiry 2001-2010.Lingua, 134:219–248. Michelle Suijkerbuijk, Zoë Prins, Marianne de Heer Kloots, Willem Zuidema, and Stefan L Frank. 2025a. Blimp-nl: A corpus of dutch minimal pairs and ac- ceptability judgments for language model evaluation. Computation...
2001
-
[2021]
Dalaj - a dataset for linguistic acceptability judgments for swedish: Format, baseline, sharing. Preprint, arXiv:2105.06681. 11 Joachim Wagner, Jennifer Foster, and Josef Genabith
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.