Pith. sign in

REVIEW 3 major objections 6 minor 26 references

Large corpora and large language models: a replicable method for automating grammatical annotation

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a supervised LLM pipeline can be trained in about an hour to annotate grammatical constructions at over 90% accuracy, making corpus-wide annotation feasible.

desk verdict A useful, reproducible LLM-annotation recipe whose headline accuracy claim is undermined by a prevalence mismatch between the eval set and the target corpus. read the letter →

arxiv 2411.11260 v2 pith:PEZFCSQW submitted 2024-11-18 cs.CL

Cameron Morin , Matti Marttinen Larsson This is my paper · ORCID
classification cs.CL
keywords corpuslinguisticsgrammaticalvariationlargelanguagemodelsautomaticannotationevaluativeconsiderconstructionpromptengineeringchain-of-thoughtpromptingEnTenTen21
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that a supervised pipeline built around a large language model can replace much of the manual sifting in corpus linguistics: with prompt engineering, iterative corrective training, and chain-of-thought classification, the model is trained to distinguish the English evaluative construction 'consider X (as) (to be) Y' from the many other uses of consider. On a held-out test of 102 sentences the pipeline reaches over 90% accuracy, with precision, recall, and Matthews correlation coefficient also reported. If the method holds, linguists can automatically annotate very large numbers of corpus tokens, here the 18 million hits for CONSIDER in EnTenTen21, instead of hand-cleaning random samples. The authors present the pipeline as replicable and adaptable to other grammatical constructions and languages, with caveats about edge cases, model choice, and sensitivity to prompt wording.

What carries the argument

The load-bearing mechanism is an iterative, conversation-scoped training loop rather than a fine-tuned model: the LLM is first shown roughly 500 hand-classified sentences and asked to reflect on the criteria, then classifies batches of 20-25 sentences and receives corrective feedback after each batch, with the refined criteria carrying over only within that chat. Chain-of-thought prompting, telling the model to reason step by step inside <thinking> tags before giving an <answer>, together with XML-wrapped examples, is what makes the feedback effective. This loop is repeated on unseen validation batches until accuracy is acceptable, and the final prompt plus transcript is the reusable artifact.

What would settle it

Re-run the published prompt transcript and training data on a fresh stratified sample of, say, 1,000 CONSIDER sentences drawn from EnTenTen21, with each sentence labelled independently by two human annotators and the majority-class baseline computed; if model accuracy on true positives does not clearly exceed the roughly 88.5% baseline, or if human disagreement is comparable to the model's error rate, the claim that the method automates reliable annotation at scale would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that an LLM, given about 500 pre-classified examples and a few rounds of corrective feedback inside one conversation, learns the envelope of variation for a grammatical construction well enough to annotate unseen corpus sentences at better than 90% accuracy. The concrete task is binary: does a given token of consider instantiate the evaluative construction, as in 'Cain is considered a master of crime fiction,' or a cognition use, as in 'please consider all classes' or 'he considers running'? The paper reports 93% accuracy on the held-out evaluation set of 102 sentences, with recall of 94.9% and a Matthews correlation coefficient of 0.86. This is offered as evidence that the bottleneck of cleaning millions of corpus hits, where only 11.5% of a random sample of CONSIDER tokens were true positives, can be automated.

Load-bearing premise

The load-bearing premise is that a 102-sentence held-out set, labelled once by the authors, accurately represents the variety and class distribution of the 18-million-token corpus, and that those labels are reliable.

Editorial extensions

If this is right

  • A linguistic construction with millions of corpus hits can be annotated in full rather than sampled, so frequency and variant analyses can be run on complete populations instead of small cleaned subsets.
  • The method's roughly 60-minute training-to-evaluation cycle makes automated annotation affordable for researchers without programming skills, since the interaction happens through a chat interface.
  • The same pipeline should transfer to other clearly defined morphosyntactic variables, provided training data match the target corpus in register, time period, and variety.
  • With the model able to classify evaluative consider at scale, the substantive questions, whether the bare, as, and to be variants differ in meaning or are changing over time, become empirically tractable.
  • Evaluation metrics such as the Matthews correlation coefficient are the appropriate guardrail for imbalanced annotation tasks, and the authors recommend reporting the full confusion matrix.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because only 11.5% of raw CONSIDER tokens in the corpus are true positives, a classifier that simply said 'no' would already score about 88.5% accuracy, so the reported accuracy needs to be read together with recall and MCC, which the paper provides.
  • Editorial inference: a direct comparison of the pipeline against two independent human annotators on the same test sentences, measuring agreement rather than only model accuracy, would clarify how much of the residual error is annotation noise rather than model failure.
  • Editorial inference: the conversation-scoped training design means the method is most reliable for well-delimited formal distinctions, and extending it to semantically or pragmatically layered phenomena would require testing whether feedback generalizes across ambiguous edge cases, as the paper itself flags.
  • Editorial inference: a natural testable extension is to measure whether the same prompt transcript performs comparably on a different large language model, which the authors partially did with a newer model version.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a supervised, prompt-based workflow in which Claude 3.5 Sonnet is iteratively trained via in-context examples and corrective feedback to classify occurrences of the English evaluative consider construction as evaluative or non-evaluative. The pipeline is described in three stages—prompt engineering, iterative training/validation, and evaluation—using the case study of 'consider X (as) (to be) Y' with data from the NOW corpus and EnTenTen21. On a held-out set of 102 sentences the authors report 93% accuracy (95/102), precision 88.1%, recall 94.9%, F1 91.4%, and MCC 0.86; a replication with Claude 3.7 Sonnet reported in footnote 5 gives similar scores. The central claim is that this performance validates the use of the pipeline for annotating the roughly 18-million-token population of CONSIDER in EnTenTen21.

Significance. The paper addresses a genuine bottleneck in corpus linguistics and has several strengths: the pipeline is described in a way that linguists without programming expertise could follow; the OSF repository contains the prompt histories and annotated data; an independent replication with a newer model is reported; and MCC is used rather than accuracy alone. The discussion of caveats covering training-data representativeness, model brittleness, legal issues, and class imbalance is thoughtful. If the reported accuracy transfers to the target corpus distribution, the method would be a practical addition to the corpus linguist's toolbox. However, the current evidence does not yet establish that transfer: the evaluation set appears to be enriched for the positive class relative to the target population, and several quantities needed to assess error rates at the operational base rate are missing. The contribution is therefore promising but not fully validated.

major comments (3)
  1. [Section 3.2 / Table 1 and Section 1] The evaluation set is not shown to be representative of the target corpus. Solving the reported metrics in Table 1 (N=102, accuracy=95/102, precision=0.881, recall=0.949) gives an approximate confusion matrix of TP=37, FP=5, FN=2, TN=58, i.e. about 38% positives in the test set. This differs sharply from the 11.5% true-positive rate measured on a random 200-token sample of EnTenTen21 in Section 1. On the test set a majority-class classifier achieves only about 62% accuracy, whereas on the target corpus it would achieve about 88.5% accuracy. The paper does not report the class balance of the evaluation set, does not give a majority-class baseline, and does not state the source corpus or sampling procedure for the 102 test sentences. The headline claim of 'over 90% accuracy' therefore does not yet establish performance on the skewed distribution that the pipeline would encounter in the 18-million-token population; the authors should add a prevalence-matched evaluation on a random corpus sample or report sensitivity and specificity at the 11.5% base rate.
  2. [Table 1 / Section 3.2] Table 1 reports only rounded aggregate metrics. For a 102-sentence test set, the 95% confidence interval for accuracy is wide (roughly 86–97%), and a single accuracy score obscures the asymmetry between false positives and false negatives that determines the cost of errors when the pipeline is applied to 18 million tokens. The authors' own Section 4 notes that imbalance makes it essential to consider all dimensions of the confusion matrix, but the paper does not provide the 2x2 table, per-class precision/recall, or confidence intervals. MCC is helpful, but a single scalar still does not communicate expected error counts at the operational prevalence. Please include the full confusion matrix and interval estimates.
  3. [Section 3.2 / Section 4] The gold standard used for evaluation is not demonstrated to be reliable. The paper reports no inter-annotator agreement, no second annotation pass, and no description of how many annotators produced the human labels for the training, validation, and test sets. The statement in Section 4 that inter-annotator disagreement is 'likely to be minimal' is an assertion rather than evidence. Since the reported accuracy and related metrics are agreement rates with this gold standard, unmeasured label noise would inflate the apparent performance of the model. I request a quantified IAA on a subset of the data or, failing that, a detailed annotation protocol including the number of annotators, the adjudication procedure, and any consistency checks.
minor comments (6)
  1. [Section 3.2] The terms 'pretraining' and 'unsupervised classification' are used in a nonstandard way for LLM workflows; consider 'in-context training' and 'unlabeled classification with feedback' to avoid confusion with model pretraining.
  2. [Section 3.2] State the total number of corrective-feedback rounds and the exact batch sizes used in each iteration, since the current description makes the amount of training implicit.
  3. [Throughout] The model is referred to as 'Claude', 'Claude 3.5 Sonnet', and 'Claude3' in different places; standardize the terminology and include the exact model version and access date in the main text.
  4. [Throughout] Use either 'EnTenTen21' or 'EnTenTen2021' consistently throughout the paper.
  5. [Footnote 5] The replication with Claude 3.7 Sonnet is reported only in a footnote; because replicability is a central claim of the paper, the replication results should appear in the main text or in an appendix.
  6. [Section 1] The reference to 'Brazaitis and Status (2023)' should be corrected to 'Brazaitis and Satas (2023)' to match the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the accuracy claim is an external measured agreement on a blind held-out set; the method's derivation does not reduce to its inputs.

full rationale

The paper's central claim, >90% accuracy on held-out test samples, is an empirical measurement against an external human-annotated gold standard rather than an analytically forced consequence of its inputs. The pipeline is explicitly empirical: approximately 500 pre-classified sentences, 100 supervised training sentences with corrective feedback, and iterative validation rounds are followed by a final blind evaluation on 102 unseen sentences, with metrics reported in Table 1 and a replication using Claude 3.7 described in footnote 5. The evaluation labels are human judgments, not outputs of the model, so the reported accuracy is not self-definitional: 'accuracy' is not defined in terms of the model's own outputs but as agreement with an independent gold standard. Iterative feedback is a form of training, and the blind held-out evaluation prevents the accuracy from being forced by construction. The paper's self-citations (e.g., Marttinen Larsson 2023; Leclercq and Morin 2023) are background references and do not carry the load of the method. The absence of class-balance reporting and the mismatch between the roughly balanced evaluation set and the corpus's 11.5% true-positive base rate is a real external-validity concern about whether 93% transfers to the operational population, but it does not make the measurement circular; the paper itself flags imbalance and recommends Matthews correlation coefficient. No step in the derivation chain equates the predicted quantity to a fitted input, a renamed variable, or a load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's pipeline depends on hand-chosen procedural parameters (training set sizes, batch sizes, number of feedback rounds) and on assumptions about gold-label correctness, sample representativeness, and the generalization of in-context feedback. No new theoretical entities are introduced.

free parameters (3)
  • Pretraining set size = ~500 sentences
    Chosen by the authors to provide initial examples; no sensitivity analysis is reported.
  • Number of sentences per training batch = 20-25
    Author-selected batch size for interactive feedback; no sensitivity analysis is reported.
  • Number of iterative feedback rounds = not reported
    Training continued until the desired accuracy was reached; the number of rounds is not stated, making the procedure hard to replicate exactly.
assumptions (3)
  • domain assumption The human gold-standard labels used for training and evaluation are correct.
    The evaluation metrics are computed against the authors' manual annotations, but inter-annotator agreement is not measured (Section 3.2).
  • domain assumption The sampled sentences are representative of the target population of 'consider' constructions in the NOW and EnTenTen21 corpora.
    The paper does not describe how the training, validation, and test sets were sampled or their class balance (Section 3.2).
  • domain assumption Claude 3.5 Sonnet's in-context 'learning' from corrective feedback reliably improves classification in a way that generalizes to unseen data.
    The method relies on iterative feedback within a single conversation (Section 3.2); the mechanism is not validated beyond the reported scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large corpora and large language models: a replicable method for automating grammatical annotation." pith.science (2026). https://pith.science/paper/PEZFCSQW

@misc{pith2026241111260,
  author       = {Pith},
  title        = {Pith review of: Large corpora and large language models: a replicable method for automating grammatical annotation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEZFCSQW}},
  note         = {Machine review of arXiv:2411.11260}
}
read the original abstract

Much linguistic research relies on annotated datasets of features extracted from text corpora, but the rapid quantitative growth of these corpora has created practical difficulties for linguists to manually annotate large data samples. In this paper, we present a replicable, supervised method that leverages large language models for assisting the linguist in grammatical annotation through prompt engineering, training, and evaluation. We introduce a methodological pipeline applied to the case study of formal variation in the English evaluative verb construction 'consider X (as) (to be) Y', based on the large language model Claude 3.5 Sonnet and corpus data from Davies' NOW and EnTenTen21 (SketchEngine). Overall, we reach a model accuracy of over 90% on our held-out test samples with only a small amount of training data, validating the method for the annotation of very large quantities of tokens of the construction in the future. We discuss the generalisability of our results for a wider range of case studies of grammatical constructions and grammatical variation and change, underlining the value of AI copilots as tools for future linguistic research, notwithstanding some important caveats.

Figures

Figures reproduced from arXiv: 2411.11260 by the authors.

Figure 1
Figure 1. illustrates the pipeline that we utilized in the present study. A key feature of our approach is that LLMs such as Claude receive substantial benefits in performance through iterative refinement. This involves the synergy between training and validation, which can be repeated until a desirable accuracy is reached. The second part of the pipeline is thus iterative, consisting of a back-and-forth training process. Fir… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [3]

    Anthropic

    https://www.anthropic.com/news/projects (accessed 15 November 2024). Anthropic. n.d. Build with Claude. https://docs.anthropic.com/en/docs/build-with-claude/ (accessed 15 November 2024). Belal, Mohammed, James She & Simon Wong

    work page 2024

  2. [11]

    Dunn, Jonathan

    https://escholarship.org/uc/item/35x5m9cm (accessed 10 March 2025). Dunn, Jonathan

    work page 2025

  3. [12]

    ArXiv https://arxiv.org/abs/2408.07215

    Can large language models reason? A characterization via 3-SAT. ArXiv https://arxiv.org/abs/2408.07215. Hekkel, Valerie, Driederike Schulz & Marta Lupica Spagnolo

    arXiv

  4. [14]

    ArXiv https://arxiv.org/abs/2404.02261 (accessed 15 November 2024)

    LLMs in the loops: Leveraging large language model annotations for active learning in low-resource languages. ArXiv https://arxiv.org/abs/2404.02261 (accessed 15 November 2024). Kilgariff, Adam, Vít Baisa, Jan Bušta, Miloš Jakubíček, Vojtěch Kovář, Jan Michelfeit, Pavel Rychlý & Vít Suchomel

    arXiv 2024

  5. [15]

    https://doi.org/10.24338/cons-

    No equivalence: A new principle of no synonymy.Constructions 15(1). https://doi.org/10.24338/cons-

    work page doi:10.24338/cons-

  6. [17]

    World Englishes

    Double modals in Australian and New Zealand English. World Englishes. Advance online publication https://doi.org/10.1111/weng.12639. Morin, Cameron & Jack Grieve

    work page doi:10.1111/weng.12639

  7. [18]

    Levshina, Natalia

    https://doi.org/10.5964/bioling.14391. Levshina, Natalia

    work page doi:10.5964/bioling.14391

  8. [19]

    PLoS One 19(1)

    The semantics, sociolinguistics, and origins of double modals in American English: New insights from social media. PLoS One 19(1). https://doi.org/10.1371/journal.pone.0295799. Ollion, Etienne, Rubing Shen, Ana Macanovic & Arnault Chatelain

    work page doi:10.1371/journal.pone.0295799

Show all 26 references
  1. [20]

    https://doi.org/10

    ChatGPT for text annotation? Mind the hype!. https://doi.org/10. 31235/osf.io/x58kn (accessed 15 November 2024). OpenAI

    2024

  2. [21]

    Chicco, Davide & Giuseppe Jurman

    https://doi.org/10.1016/j.iswa.2024.200336. Chicco, Davide & Giuseppe Jurman

    2024

  3. [22]

    https://chat.openai.com (accessed 15 November 2024)

    ChatGPT-4, version 4.0 [Large language model]. https://chat.openai.com (accessed 15 November 2024). OpenAI & Ekin, Sabit

    2024

  4. [23]

    36227/techrxiv.22683919.v2 (accessed 15 November 2024)

    Prompt engineering for ChatGPT: A quick guide to techniques, tips, and best practices.TechRxiv https://doi.org/10. 36227/techrxiv.22683919.v2 (accessed 15 November 2024). Piantadosi, Steven

    2024

  5. [24]

    ArXiv https://doi.org/10.48550/arXiv.2310.20689 (accessed 15 November 2024)

    Learning from mistakes makes LLMs better reasoner. ArXiv https://doi.org/10.48550/arXiv.2310.20689 (accessed 15 November 2024). Torrent, Tiago Timponi, Thomas Ho ffmann, Arthur Lorenzi Almeida & Mark Turner

    2024 doi arXiv

  6. [25]

    In Bouamor Houda, Juan Pino & Kalika Bali (eds.)

    Counting the bugs in ChatGPT’s wugs: A multilingual investigation into the morphological capabilities of a large language model. In Bouamor Houda, Juan Pino & Kalika Bali (eds.). Proceedings of the 2023 Conference on empirical Methods in natural language processing , 6508–

    2023

  7. [26]

    ArXiv https://doi.org/10.48550/arXiv.2302.11382 (accessed 15 November 2024)

    A prompt pattern catalog to enhance prompt engineering with ChatGPT. ArXiv https://doi.org/10.48550/arXiv.2302.11382 (accessed 15 November 2024). Yu, Danni, Luyang Li, Hang Su & Matteo Fuoli

    2024 doi arXiv

  8. [46]

    Consider

    Britannica Dictionary. n.d. Ask the editor: “Consider” and “consider as”. https://www.britannica.com/dictionary/eb/qa/consider-and- consider-as (accessed 25 February 2025). Caruccio, Loredana, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugan Sundaramurthy & ...

    2025

  9. [2008]

    https://www.english-corpora.org/coca/ (accessed 15 November 2024)

    The Corpus of Contemporary American English (COCA). https://www.english-corpora.org/coca/ (accessed 15 November 2024). Davies, Mark

    2024

  10. [2016]

    https://www.english-corpora.org/now/(accessed 15 November 2024)

    The NOW corpus (news on the web). https://www.english-corpora.org/now/(accessed 15 November 2024). Denning, Joseph, Xiaohan Guo, Bryor Snefjella & Idan Blank

    2024

  11. [2017]

    https://spacy.io/(accessed 10 March 2025)

    spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks, and incremental parsing. https://spacy.io/(accessed 10 March 2025). Jacques, Guillaume

    2025

  12. [2018]

    https:// www.qcaa.qld.edu.au/downloads/senior-qce/common/snr_glossary_cognitive_verbs.pdf (accessed 6 February 2025)

    Glossary of cognitive verbs: General syllabuses: Queensland Government. https:// www.qcaa.qld.edu.au/downloads/senior-qce/common/snr_glossary_cognitive_verbs.pdf (accessed 6 February 2025). Shengnan, An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou & Weizhu Chen

    2025

  13. [2020]

    BMC Genomics 21(6)

    The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21(6). https://doi.org/10.1186/s12864-019-6413-7. Coats, Steven

    doi

  14. [2022]

    InProceedings of the 36th international conference on neural information processing systems, 24824– 24837

    Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th international conference on neural information processing systems, 24824– 24837. Red Hook, NY: Curran Associates. https://dl.acm.org/doi/10.5555/3600270.3602070. Weissweiler, Leoni...

  15. [2023]

    Bender, Emily & Alexander Koller

    Leveraging ChatGPT as text annotation tool for sentiment analysis.ArXiv https://doi.org/ 10.48550/arXiv.2306.17177. Bender, Emily & Alexander Koller

    doi arXiv

  16. [2024]

    LREC-COLING 2024 6504–

    Evaluating the quality of a corpus annotation scheme using pretrained language models. LREC-COLING 2024 6504–

    2024

  17. [5198]

    https:// aclanthology.org/2020.acl-main.463/(accessed 10 March 2025)

    Association for Computational Linguistics. https:// aclanthology.org/2020.acl-main.463/(accessed 10 March 2025). Bonilla, Johnatan Esteban Huerfano, Laura M. Merino Hernández & Larsson Matti Marttinen. In press (2025). BERT ’s interpretation of literalmente ‘literally’: What d...

    2025

  18. [6514]

    Anthropic

    Turin: European Language Resources Association (ELRA) https://aclanthology.org/2024.lrec-main.577.pdf (accessed 10 March 2025). Anthropic. 2024a. Claude 3.5 Sonnet [large language model]. https://claude.ai/ (accessed 15 November 2024). Anthropic. 2024b. Collaborate with Claude...

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.