Automatic Annotation of Ancient Greek Vowel Length

Albin Th\"orn Cleland; Eric Cullhed

REVIEW 2 major objections 4 minor 15 references

Automatic Annotation of Ancient Greek Vowel Length

T0 review · 2 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The authors build the first general-purpose macronizer for Ancient Greek and show that a character-level transformer trained on its silver output generalizes beyond it, reaching 92% accuracy on a gold benchmark and improving scansion.

desk verdict Useful resources and a clear pipeline, but the headline accuracy figure is probably in-sample and cannot carry the generalization claim. read the letter →

arxiv 2608.01935 v1 pith:JEFNQZTH submitted 2026-08-03 cs.CL cs.AIcs.LG

Albin Thörn Cleland , Eric Cullhed This is my paper

classification cs.CLcs.AIcs.LG

keywords AncientGreekvowellengthdichronamacronizationcharacter-leveltransformersilvertrainingdatasequencelabelingdigitalphilology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ancient Greek manuscripts do not mark whether α, ι, and υ are long or short, yet this 'dichronic' distinction matters for grammar, meaning, and metre. The authors construct the first general-purpose macronizer: a rule-based pipeline that combines dictionaries, accentuation laws, and recursive normalization to mark vowel length in annotated input, resolving 69.2% of ambiguous open syllables across a forty-million-token corpus. The paper's central claim is that a small character-level transformer trained on this pipeline's output, with abstentions excluded from the loss, learns to generalize beyond the rules: it reaches 92.0% accuracy and 92.2% F1 on a gold benchmark covering 99.7% of ambiguous positions, and its macronizations improve hexameter scansion accuracy by about 5.8 percentage points. The result matters because no public macronized Greek corpus existed at scale, and this enables downstream prosodic, syntactic, and stylistic analysis.

What carries the argument

The central objects are the dichrona—α, ι, and υ, letters whose phonemic length is context-dependent—and the silver-training protocol. The rule-based macronizer resolves them with underscore (long) and caret (short) marks through thirteen modules: database lookups, Attic accentuation laws, and recursive normalization (recovering oblique cases, undoing elision, handling enclitics). The transformer's decisive mechanism is ignore-index masking: positions where the teacher abstained are excluded from the loss, so the student learns only from confident teacher judgments. This turns the teacher's conservatism into an asset, allowing the student to cover nearly all ambiguous positions while inherit

What would settle it

Compare the 1,916 benchmark lines against the forty-million-token development corpus (including the verse, prose, and patristic subsets); if any benchmark line or a near-identical variant occurs there, the evaluation is in-sample and the 92% figure does not support the generalization claim. A decisive test would re-run the evaluation on a held-out set of texts from different periods and dialects not present in the development corpus.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a student model can surpass a conservative rule-based teacher if the teacher's abstentions are treated as missing labels rather than as implicit 'short'. Using a character-level transformer with two embedding planes (one for letters, one for diacritics) and a three-way long/short/none head, the authors train on 2.23 million silver-annotated sentences, masking from the loss every position the rule-based macronizer left unmarked. On the manually annotated benchmark of 1,916 open-syllable dichrona, the student reaches 92.0% mixed-label accuracy and 92.2% F1 while covering 99.7% of positions—a gain in F1 of 6.3 points on the difficult long label and a 9.1-po

Load-bearing premise

The benchmark texts were chosen not to coincide with the verse corpus used in rule development, but the same classical authors also appear in the forty-million-token corpus used for development and transformer training; if benchmark lines appear there, the generalization claim rests on circular evidence.

Editorial extensions

If this is right

A stand-alone macronizer now exists for arbitrary Ancient Greek input carrying lemma, part-of-speech, and morphology, so any CoNLL-U-annotated corpus can be macronized without bespoke dictionaries.
The released macronized corpus—2.23 million sentences with 69.2% of open-syllable dichrona resolved—provides the first large-scale training data for vowel-length-sensitive Greek NLP.
The transformer's 99.7% coverage means macronization can be applied end-to-end to verse and prose, not just to the subset the rules confidently handle.
Macronized input improves a downstream hexameter scansion classifier by a mean 5.8 percentage points, suggesting similar gains for other prosodic or phonological tasks.
The recipe of training a small student on a rule-based teacher's output with abstentions masked from the loss is transferable to other long-tail annotation problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the benchmark lines overlap with the development corpus used for rule tuning and transformer training, the reported 92% accuracy would be an in-sample estimate; a truly disjoint evaluation could be lower.
The deliberate Attic normalization means the macronizer systematically biases non-Attic texts (Homeric, Doric, Hellenistic, Byzantine) toward Attic lengths; adapting to metadata could correct this.
Because the student's ceiling is the teacher's ceiling, the current results likely understate what could be achieved with even a small amount of manually corrected fine-tuning data.
The same 'abstention-masked silver training' paradigm could accelerate annotation for other under-resourced languages with vowel-length or tone contrasts, such as Latin, Icelandic, or Cantonese.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

This paper does something new: it builds a first general-purpose macronizer for Ancient Greek, releases a 2.23M-sentence silver corpus, a character transformer, and a manually annotated benchmark. The rule-based modules are described in enough detail to be reproduced, and the limitation section is unusually candid about Attic normalization and the teacher ceiling. Credit is due for the resources alone.

The problem is the evaluation. The transformer's 92% F1 on the Norma benchmark is presented as evidence that it generalizes past positions the rule-based system leaves unmarked. But Norma draws on Aeschylus, Plato, and others whose works are standard components of Perseus and First1KGreek, which are exactly the corpora that make up OGA, the training source. The paper only excludes Hypotactic texts from the benchmark, not OGA. That means the silver labels for those very lines were likely in the training data. The high accuracy could be memorization or teacher-label copying, not generalization. This is a load-bearing flaw: without an out-of-sample benchmark, RQ2 is unanswered. The stress-test note is correct.

Two smaller issues. First, the rule-based system's mixed F1 (78.7) is lower than the constant-short baseline (83.1) on its own covered positions. The paper frames it as 'conservative,' but being worse than always predicting short deserves an explicit explanation. Second, the scansion result reports a 5.8% relative improvement but no absolute accuracy, so we cannot tell whether macronization makes a terrible system adequate or a good system slightly better.

None of this is fatal to the paper as a resource. The corpus and models are valuable even if the empirical claims need rework. But the generalization claim should be stated more cautiously until the benchmark is confirmed out-of-sample, or a fresh benchmark is collected. A serious referee should ask for exactly that, plus the scansion baseline and a discussion of the rule-based baseline failure.

I would send this to review. It is worth the time, and the fixes are concrete.

Referee Report

2 major / 4 minor

Summary. The paper presents a two-stage pipeline for macronizing Ancient Greek dichrona. A rule-based macronizer, built from Wiktionary-derived dictionaries, LSJ, Hypotactic metrical data, custom rules, and recursive normalization, is run over the 40M-token OGA corpus to produce silver training labels. A small character-level transformer is then trained on those labels, with teacher abstentions masked from the loss. The transformer is evaluated on a new manually annotated benchmark, Norma Syllabarum Graecarum, where it is reported to reach 92.0% mixed-label accuracy and 92.2% mixed-label F1 while covering 99.7% of ambiguous positions, compared with 66.7% accuracy and 78.7% F1 at 69.5% coverage for the rule-based system. The paper also reports that macronization improves a downstream hexameter-scansion classifier by about 5.8% absolute accuracy.

Significance. If the reported numbers are valid and the benchmark is truly out-of-sample, the paper would deliver the first general-purpose Ancient Greek macronizer, a large silver-annotated corpus, a lightweight character-level transformer, and an open manually annotated benchmark. These are substantial resources for a low-resource historical language, and the paper makes them publicly available. The downstream scansion experiment is a useful demonstration of practical value. However, the central generalization claim depends on the Norma benchmark being independent of the OGA corpus used to generate the transformer's silver training data, and the paper does not establish that independence.

major comments (2)

[§4, footnote 9; §2-3] The paper's main claim that the transformer 'generalizes past the cases the rule-based system leaves unmarked' rests on the Norma benchmark being out-of-sample for the transformer's silver training corpus. Footnote 9 states only that the Norma texts were chosen not to coincide with Hypotactic as of 2025. But the transformer was trained on OGA (Section 3), and OGA is a combination of First1KGreek, PatristicTextArchive, and Perseus canonical-greekLit (Section 2). Many Norma authors (Aeschylus, Sophocles, Euripides, Aristophanes, Plato, Thucydides, etc.) are standard components of those collections. If Norma lines occur in OGA, the transformer may have been evaluated on silver labels it saw during training, making the 92.0/92.2 numbers consistent with memorization or label copying rather than generalization. The paper should report the overlap between Norma and OGA, exclude any overlapping
[§4, Table 1] The rule-based system's mixed F1 (78.7) is reported to be below the constant-short baseline's mixed F1 (83.1) even though the rule-based system covers only 69.5% of positions. This is surprising for a 'reliable but conservative' teacher. The comparison is not apples-to-apples unless the baseline and the rule-based system are evaluated on exactly the same covered positions. The paper should clarify whether accuracy/F1 for a system are computed only over positions it covers, and should report the constant-short baseline restricted to the rule-based system's covered positions. If the rule-based teacher is substantially worse than the baseline even on its own predictions, the claim that the transformer learns from a trustworthy teacher needs more scrutiny.

minor comments (4)

[§4, after Table 1] Typo: 'tranformer' should be 'transformer'.
[Limitations] Typo: 'vowel lenghts' should be 'vowel lengths'.
[§4, footnote 9] Footnote 9 is a URL rather than a proper citation; the relationship between Norma and OGA should be stated in the main text, not left to a footnote.
[§4, Table 1] Parenthetical 'Long (0.0)...' entries for the constant-short baseline are hard to read; consider using em-dashes or a sentence in the caption explaining that precision/recall/F1 are undefined when no long predictions are made.

Circularity Check

1 steps flagged · score 6.0 of 10

Norma benchmark may be in-sample: benchmark texts drawn from OGA collections while only Hypotactic is excluded

fitted input called prediction [Section 4 'Benchmarking' (footnote 9), with Section 2 ('OGA') and Section 3 ('Silver training data')]
"All the texts were chosen so as not to coincide with those included in Hypotactic as of 2025."

The benchmark's only stated exclusion is Hypotactic, a scanned-verse database read by the rule-based module. The transformer, however, is trained on the OGA corpus: Section 3 says 'We ran it over the forty-million-token OGA corpus', and Section 2 defines OGA as 'a combination of First1KGreek, PatristicTextArchive, and Perseus’ canonical-greekLit.' Norma's authors (Aeschylus, Sophocles, Euripides, Aristophanes, Thucydides, Plato, Plutarch, etc.) are canonical works in exactly those collections. If any Norma line occurs in OGA, the transformer saw silver labels for that line during training, so its 92.0/92.2 accuracy/F1 and 99.7% coverage on Norma are in-sample measurements, not evidence of generalization to unseen cases. The claimed ability to 'generalize past the cases the rule-based syste

full rationale

The paper is otherwise largely self-contained: the rule-based macronizer is built from dictionaries and algorithmic modules, and training a transformer on silver labels with abstentions masked from the loss is a legitimate distillation setup. The Limitations section appropriately concedes that 'the ceiling it approaches is the teacher’s ceiling rather than the gold standard’s,' which limits expectations but is not itself circular. The serious issue is the evaluation. The benchmark section excludes only Hypotactic lines, but Hypotactic is merely one database used by the rule-based module; the transformer's training data are the OGA corpus, composed of First1KGreek, PatristicTextArchive, and Perseus canonical-greekLit. Norma's authors are standard classical authors whose works are components of those collections. Unless a further, unstated exclusion removes all overlapping lines, the transformer has seen silver labels for the benchmark texts during training, making its 92.0/92.2/99.7 numbers an in-sample fit rather than an out-of-sample generalization. Because the transformer's result is the sole support for RQ2 (the rule-based mixed F1 is 78.7, below the constant-short baseline of 83.1), this potential leakage materially undermines the central claim. No other circular step was found: the scansion experiment uses a held-out split, and the self-citations (Thörn Cleland 2024, 2025) are data or prior-work references, not load-bearing justifications.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest on two external taggers, hand-built databases from Wiktionary, LSJ, and Hypotactic, an Attic-normalization policy, a short-default prior, and the independence of the gold benchmark from the development and training corpus. The benchmark independence is not demonstrated and is the most fragile assumption.

free parameters (1)

short-default prior for ambiguous forms (e.g., ἀεί) = implicit prior of ~0.7 short
Section 2: 'around seven out of ten dichrona in the Attic classics are short: defaulting such forms to short performs better on average.' Used to decide forms with no clear genre or period fault line.

assumptions (5)

domain assumption Accentuation laws of classical Attic (the two halves of the σωτῆρ rule and short ultima of proparoxytones) constrain vowel length practically without exception.
Section 2, Algorithmic modules; the rules are assumed correct for the target variety and are a load-bearing part of the rule-based system.
domain assumption Attic is the appropriate normalization for all ambiguous forms.
Section 2: ambiguous tokens are 'assigned the ancient Attic length across the board'; the limitations admit this is wrong by design for non-Attic texts.
domain assumption The Norma benchmark texts are independent of the OGA corpus used for development and training.
Section 4; only exclusion from Hypotactic is stated, yet the benchmark's classical authors are likely represented in OGA's Perseus and First1KGreek components.
domain assumption Lemma, POS, and morphology annotation from odyCy and Trankit is accurate enough for vowel-length decisions.
Section 2 and Limitations; the paper reports tagger scores (POS 95-96, LEMMA 83-91) but does not quantify how much annotation error propagates into macronization.
domain assumption Manually annotated gold lengths in Norma are correct.
Section 4; the benchmark is new and introduces 1,916 gold positions, but no inter-annotator agreement is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automatic Annotation of Ancient Greek Vowel Length." pith.science (2026). https://pith.science/paper/JEFNQZTH

@misc{pith2026260801935,
  author       = {Pith},
  title        = {Pith review of: Automatic Annotation of Ancient Greek Vowel Length},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEFNQZTH}},
  note         = {Machine review of arXiv:2608.01935}
}

read the original abstract

Prior work in Ancient Greek NLP relies on corpora that do not disambiguate the phonemic vowel length of alpha, iota, and ypsilon, together known as the dichrona. Depending on lexeme, morphology, sandhi, syntax, and conventions of period, genre, and verse form, each of these letters can represent either a long or a short vowel. Deciding and marking the correct length is known as "macronizing", a long-tail problem given the sheer mass of word forms and the context dependency of individual instances. No macronized corpus of Ancient Greek is publicly available at scale, so a stand-alone macronizer is needed. While previous work has shown how to build a static, corpus-bespoke vowel-length dictionary, the present paper constructs the first general-purpose macronizer for arbitrary Ancient Greek input. Given input carrying lemma, part-of-speech, and morphological annotation in the standard CoNLL-U format, a set of recursive modules lets less common word forms inherit markup from more common forms of the same lexical word. The macronizer's chief application is generating training data for machine learning: we show that a small character-level transformer trained on the macronizer's own output learns to generalize past the cases the rule-based system leaves unmarked, matching or exceeding its accuracy on a gold-standard, manually annotated benchmark of verse and prose. We also show that macronization can improve downstream prosodical NLP tasks like verse scansion.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages

[1]

Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando de Freitas. 2022. https://doi.org/10.1038/s41586-022-04448-z Restoring and attributing ancient texts using deep neural networks . Nature, 603(7900):280--283

doi:10.1038/s41586-022-04448-z 2022
[2]

Giuseppe G. A. Celano. 2024. https://doi.org/10.48550/arXiv.2410.12055 A state-of-the-art morphosyntactic parser and lemmatizer for Ancient Greek

work page Pith review arXiv doi:10.48550/arxiv.2410.12055 2024
[3]

Giuseppe G. A. Celano. 2026. https://doi.org/10.11588/DCO.2026.12.112290 Opera Graeca Adnotata : Building a 40m+ token multilayer corpus for Ancient Greek . Bd. 12:46--60 Seiten

work page doi:10.11588/dco.2026.12.112290 2026
[4]

David Chamberlain. 2023. https://hypotactic.com/ hypotactic.com

2023
[5]

Marton Kardos and Jan Kostkan. 2023. https://centre-for-humanities-computing.github.io/odyCy odyCy

2023
[6]

Jingxin Luo, Vivian Guo Li, and Peggy Pik Ki Mok. 2019. https://doi.org/10.1177/0023830919879471 The perception of cantonese vowel length contrast by mandarin speakers . Language and Speech, 63(3):635--659

work page doi:10.1177/0023830919879471 2019
[7]

David Odden. 2011. https://doi.org/10.1002/9781444335262.wbctp0020 The representation of vowel length . In Marc van Oostendorp, Colin J. Ewen, Elizabeth Hume, and Keren Rice, editors, The Blackwell Companion to Phonology. Wiley-Blackwell

work page doi:10.1002/9781444335262.wbctp0020 2011
[8]

Aglae Pizzone. 2024. https://doi.org/10.54103/interfaces-11-04 The occasionality of Byzantine didacticism: a case study from the twelfth century ( Milan , Veneranda Biblioteca Ambrosiana , c 222 inf. f. 218r) . Interfaces: A Journal of Medieval European Literatures, 11:51--73

work page doi:10.54103/interfaces-11-04 2024

Show all 15 references

[9]

Philomen Probert. 2010. Phonology. In Egbert J. Bakker, editor, A Companion to the Greek Language, pages 85--103. Wiley-Blackwell

2010
[10]

Caitlin Laura Richter, Kolbr \'u n Fri riksd \'o ttir, Korm \'a kur Logi Bergsson, Erik Anders Maher, Ragnhei ur Mar \'i a Benediktsd \'o ttir, and Jon Gudnason. 2025. https://aclanthology.org/2025.nodalida-1.56/ Assessed and annotated vowel lengths in spoken Icelandic sentenc...

2025
[11]

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108

2019 arXiv
[12]

and David Fifield

Stephen Sansom, A. and David Fifield. 2023. SEDES : Metrical position in Greek hexameter. Digital Humanities Quarterly, 17(2)

2023
[13]

Albin Thörn Cleland. 2024. A macronizer for Ancient Greek . Master's thesis, Uppsala universitet

2024
[14]

Albin Thörn Cleland. 2025. https://doi.org/10.5281/zenodo.16603568 Hidden choral stimuli: The role of accent in the refrains of Aristophanes

2025 doi
[15]

Johan Winge. 2015. Automatic annotation of Latin vowel length. Master's thesis, Uppsala University, Uppsala

2015

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag, and Nando de Freitas. 2022. https://doi.org/10.1038/s41586-022-04448-z Restoring and attributing ancient texts using deep neural networks . Nature, 603(7900):280--283

doi:10.1038/s41586-022-04448-z 2022

[2] [2]

Giuseppe G. A. Celano. 2024. https://doi.org/10.48550/arXiv.2410.12055 A state-of-the-art morphosyntactic parser and lemmatizer for Ancient Greek

work page Pith review arXiv doi:10.48550/arxiv.2410.12055 2024

[3] [3]

Giuseppe G. A. Celano. 2026. https://doi.org/10.11588/DCO.2026.12.112290 Opera Graeca Adnotata : Building a 40m+ token multilayer corpus for Ancient Greek . Bd. 12:46--60 Seiten

work page doi:10.11588/dco.2026.12.112290 2026

[4] [4]

David Chamberlain. 2023. https://hypotactic.com/ hypotactic.com

2023

[5] [5]

Marton Kardos and Jan Kostkan. 2023. https://centre-for-humanities-computing.github.io/odyCy odyCy

2023

[6] [6]

Jingxin Luo, Vivian Guo Li, and Peggy Pik Ki Mok. 2019. https://doi.org/10.1177/0023830919879471 The perception of cantonese vowel length contrast by mandarin speakers . Language and Speech, 63(3):635--659

work page doi:10.1177/0023830919879471 2019

[7] [7]

David Odden. 2011. https://doi.org/10.1002/9781444335262.wbctp0020 The representation of vowel length . In Marc van Oostendorp, Colin J. Ewen, Elizabeth Hume, and Keren Rice, editors, The Blackwell Companion to Phonology. Wiley-Blackwell

work page doi:10.1002/9781444335262.wbctp0020 2011

[8] [8]

Aglae Pizzone. 2024. https://doi.org/10.54103/interfaces-11-04 The occasionality of Byzantine didacticism: a case study from the twelfth century ( Milan , Veneranda Biblioteca Ambrosiana , c 222 inf. f. 218r) . Interfaces: A Journal of Medieval European Literatures, 11:51--73

work page doi:10.54103/interfaces-11-04 2024