{"id":"ba78981e-c201-4c6e-a231-5da4b4d4efcf","arxiv_id":"2411.11260","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An iterative prompt engineering pipeline with Claude 3.5 Sonnet classified evaluative versus non-evaluative 'consider' sentences at 93% accuracy on 102 held-out sentences.","lead":"A research team showed that a commercial AI, Claude 3.5 Sonnet, can be prompted and iteratively trained to tag corpus sentences containing the verb 'consider' as evaluative or not, reaching about 93% accuracy on a small held-out test. The paper offers this as a reusable pipeline for scaling grammatical annotation, but the small test set, missing baseline, and lack of error bars make the strength of the claim uncertain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy is on an eval set that Table 1 implies is ~38% positive, not the 11.5% base rate of the target corpus; corpus-scale validation therefore needs a prevalence-matched test.","rationale":"The paper is an honest, useful methodological case study, and the OSF materials plus the Claude 3.7 replication are real evidence. My concern is not that the model is near a majority baseline in the reported evaluation; in fact, with ~38% positives, the baseline is only ~62% and the model is well above it. The problem is representativeness: the metrics in Table 1 cannot arise from a random sample of the target corpus, so the 93% figure does not by itself license the 'annotate very large quantities' claim. The reader's weakest_assumption correctly focused on class balance and the 11.5% base rate, but the specific arithmetic strengthens the point: the eval set appears enriched, making the majority-baseline objection to the eval set weaker but the extrapolation problem more acute. What is needed is a corpus-representative evaluation with reported class balance and inter-annotator agreement. Since the authors' own guidelines emphasize reporting confusion-matrix dimensions (Section 4), this is a natural and feasible addition. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":11358,"tokens_out":6779,"duration_ms":68528,"concrete_test":"Reconstruct the confusion matrix from the OSF data and report counts/class balance; then run the exact final prompt on a fresh random sample of 200 CONSIDER tokens from EnTenTen21 (expected ~23 positives), double-coded by two annotators. If precision/recall on this corpus-representative sample stays close to Table 1 and clearly beats the majority-class baseline, the concern is resolved; if performance drops, the claim of validation for very large quantities is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that >90% accuracy on held-out samples validates using the pipeline to annotate the 18M-token CONSIDER population. The load-bearing condition is that the 102-sentence eval set resembles that population. It does not: solving from Table 1 (N=102, 7 errors, precision 0.881, recall 0.949) gives ~39 positives in the eval set, i.e. ~38% prevalence, versus the 11.5% true-positive rate measured on a random 200-token sample in Section 1. The Claude 3.7 replication in footnote 5 implies the same ~38% prevalence. Thus the reported accuracy is measured on a roughly balanced sample, where the majority-class baseline is ~62%, not on the skewed corpus distribution where a trivial classifier is 88.5% accurate. The paper never reports class balance, so the reader cannot tell whether the 93% transfers to the real task; at the operational base rate, precision and recall could be much lower even if they are good in the enriched eval set. The absence of class balance is therefore not a cosmetic omission: it is the quantity that connects the experiment to the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a supervised, prompt-based workflow in which Claude 3.5 Sonnet is iteratively trained via in-context examples and corrective feedback to classify occurrences of the English evaluative consider construction as evaluative or non-evaluative. The pipeline is described in three stages—prompt engineering, iterative training/validation, and evaluation—using the case study of 'consider X (as) (to be) Y' with data from the NOW corpus and EnTenTen21. On a held-out set of 102 sentences the authors report 93% accuracy (95/102), precision 88.1%, recall 94.9%, F1 91.4%, and MCC 0.86; a replication with Claude 3.7 Sonnet reported in footnote 5 gives similar scores. The central claim is that this performance validates the use of the pipeline for annotating the roughly 18-million-token population of CONSIDER in EnTenTen21.","tokens_in":11562,"tokens_out":14700,"duration_ms":139459,"significance":"The paper addresses a genuine bottleneck in corpus linguistics and has several strengths: the pipeline is described in a way that linguists without programming expertise could follow; the OSF repository contains the prompt histories and annotated data; an independent replication with a newer model is reported; and MCC is used rather than accuracy alone. The discussion of caveats covering training-data representativeness, model brittleness, legal issues, and class imbalance is thoughtful. If the reported accuracy transfers to the target corpus distribution, the method would be a practical addition to the corpus linguist's toolbox. However, the current evidence does not yet establish that transfer: the evaluation set appears to be enriched for the positive class relative to the target population, and several quantities needed to assess error rates at the operational base rate are missing. The contribution is therefore promising but not fully validated.","major_comments":[{"comment":"The evaluation set is not shown to be representative of the target corpus. Solving the reported metrics in Table 1 (N=102, accuracy=95/102, precision=0.881, recall=0.949) gives an approximate confusion matrix of TP=37, FP=5, FN=2, TN=58, i.e. about 38% positives in the test set. This differs sharply from the 11.5% true-positive rate measured on a random 200-token sample of EnTenTen21 in Section 1. On the test set a majority-class classifier achieves only about 62% accuracy, whereas on the target corpus it would achieve about 88.5% accuracy. The paper does not report the class balance of the evaluation set, does not give a majority-class baseline, and does not state the source corpus or sampling procedure for the 102 test sentences. The headline claim of 'over 90% accuracy' therefore does not yet establish performance on the skewed distribution that the pipeline would encounter in the 18-million-token population; the authors should add a prevalence-matched evaluation on a random corpus sample or report sensitivity and specificity at the 11.5% base rate.","section":"Section 3.2 / Table 1 and Section 1"},{"comment":"Table 1 reports only rounded aggregate metrics. For a 102-sentence test set, the 95% confidence interval for accuracy is wide (roughly 86–97%), and a single accuracy score obscures the asymmetry between false positives and false negatives that determines the cost of errors when the pipeline is applied to 18 million tokens. The authors' own Section 4 notes that imbalance makes it essential to consider all dimensions of the confusion matrix, but the paper does not provide the 2x2 table, per-class precision/recall, or confidence intervals. MCC is helpful, but a single scalar still does not communicate expected error counts at the operational prevalence. Please include the full confusion matrix and interval estimates.","section":"Table 1 / Section 3.2"},{"comment":"The gold standard used for evaluation is not demonstrated to be reliable. The paper reports no inter-annotator agreement, no second annotation pass, and no description of how many annotators produced the human labels for the training, validation, and test sets. The statement in Section 4 that inter-annotator disagreement is 'likely to be minimal' is an assertion rather than evidence. Since the reported accuracy and related metrics are agreement rates with this gold standard, unmeasured label noise would inflate the apparent performance of the model. I request a quantified IAA on a subset of the data or, failing that, a detailed annotation protocol including the number of annotators, the adjudication procedure, and any consistency checks.","section":"Section 3.2 / Section 4"}],"minor_comments":[{"comment":"The terms 'pretraining' and 'unsupervised classification' are used in a nonstandard way for LLM workflows; consider 'in-context training' and 'unlabeled classification with feedback' to avoid confusion with model pretraining.","section":"Section 3.2"},{"comment":"State the total number of corrective-feedback rounds and the exact batch sizes used in each iteration, since the current description makes the amount of training implicit.","section":"Section 3.2"},{"comment":"The model is referred to as 'Claude', 'Claude 3.5 Sonnet', and 'Claude3' in different places; standardize the terminology and include the exact model version and access date in the main text.","section":"Throughout"},{"comment":"Use either 'EnTenTen21' or 'EnTenTen2021' consistently throughout the paper.","section":"Throughout"},{"comment":"The replication with Claude 3.7 Sonnet is reported only in a footnote; because replicability is a central claim of the paper, the replication results should appear in the main text or in an appendix.","section":"Footnote 5"},{"comment":"The reference to 'Brazaitis and Status (2023)' should be corrected to 'Brazaitis and Satas (2023)' to match the reference list.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"Promising methods paper with incomplete empirical support. The missing analyses—prevalence-matched evaluation, confusion matrix, confidence intervals, and inter-annotator agreement—are all feasible with the existing OSF data, so a revision is appropriate. I see no issues with novelty, citation practice, or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a genuinely useful methodological case study, and the authors have done several things right. They ship the prompt histories, data, and annotations on OSF; they report precision, recall, F1, and MCC; and they replicated the whole training with Claude 3.7, getting nearly identical scores. That is real evidence of reproducibility, and it deserves credit.\n\nWhat is actually new here is modest but real: a concrete, chat-based pipeline for iteratively training an LLM to classify a grammatical construction, with the specific demonstration on evaluative 'consider'. The prompting principles are not novel, and the paper cites prior LLM-assisted annotation work, so the novelty is in the recipe and the case study rather than in any conceptual breakthrough.\n\nThe soft spot is the one the stress-test flags, and it is not cosmetic. The headline claim is that >90% accuracy on held-out samples validates using the pipeline on the 18M-token CONSIDER population. But Table 1 implies the eval set has roughly 38% true positives; solving from precision 0.881 and recall 0.949 over 102 sentences with 7 errors gives about 39 positives. The corpus base rate is 11.5%, as the paper itself notes. On a balanced eval set, a trivial majority-class classifier would already get 62% accuracy; on the actual corpus distribution, it would get 88.5%. So the reported accuracy does not transfer automatically, and the paper never reports class balance or a baseline comparison. This is the load-bearing missing piece for the central claim.\n\nThat said, the flaw is fixable rather than fatal. The MCC of 0.86 and the replication are reassuring, and the method is clearly not circular: the gold labels are external and manually produced. The absence of confidence intervals and inter-annotator agreement is a standard criticism but minor here given the clear-cut formal task.\n\nWho is this for? Corpus linguists who want a template for LLM-assisted annotation of a low-precision construction. They will learn more from the prompt histories than from the abstract. I would send it to peer review, and I would recommend the authors add a prevalence-matched test set and a baseline before relying on the scale-up claim.","headline":"A useful, reproducible LLM-annotation recipe whose headline accuracy claim is undermined by a prevalence mismatch between the eval set and the target corpus.","tokens_in":12107,"tokens_out":2690,"would_cite":true,"duration_ms":24911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a supervised LLM pipeline can be trained in about an hour to annotate grammatical constructions at over 90% accuracy, making corpus-wide annotation feasible.","keywords":["corpus linguistics","grammatical variation","large language models","automatic annotation","evaluative consider construction","prompt engineering","chain-of-thought prompting","EnTenTen21"],"falsifier":"Re-run the published prompt transcript and training data on a fresh stratified sample of, say, 1,000 CONSIDER sentences drawn from EnTenTen21, with each sentence labelled independently by two human annotators and the majority-class baseline computed; if model accuracy on true positives does not clearly exceed the roughly 88.5% baseline, or if human disagreement is comparable to the model's error rate, the claim that the method automates reliable annotation at scale would be falsified.","tokens_in":11136,"feed_emoji":"🤖","tokens_out":9584,"duration_ms":80578,"temperature":0.7,"pith_summary":"This paper aims to establish that a supervised pipeline built around a large language model can replace much of the manual sifting in corpus linguistics: with prompt engineering, iterative corrective training, and chain-of-thought classification, the model is trained to distinguish the English evaluative construction 'consider X (as) (to be) Y' from the many other uses of consider. On a held-out test of 102 sentences the pipeline reaches over 90% accuracy, with precision, recall, and Matthews correlation coefficient also reported. If the method holds, linguists can automatically annotate very large numbers of corpus tokens, here the 18 million hits for CONSIDER in EnTenTen21, instead of hand-cleaning random samples. The authors present the pipeline as replicable and adaptable to other grammatical constructions and languages, with caveats about edge cases, model choice, and sensitivity to prompt wording.","feed_headline":"LLM pipeline labels 'consider' constructions at 90% accuracy","feed_subtitle":"A 60-minute supervised prompting loop lets linguists classify huge corpora without hand-sifting millions of tokens.","key_machinery":"The load-bearing mechanism is an iterative, conversation-scoped training loop rather than a fine-tuned model: the LLM is first shown roughly 500 hand-classified sentences and asked to reflect on the criteria, then classifies batches of 20-25 sentences and receives corrective feedback after each batch, with the refined criteria carrying over only within that chat. Chain-of-thought prompting, telling the model to reason step by step inside <thinking> tags before giving an <answer>, together with XML-wrapped examples, is what makes the feedback effective. This loop is repeated on unseen validation batches until accuracy is acceptable, and the final prompt plus transcript is the reusable artifact.","core_discovery":"The central claim is that an LLM, given about 500 pre-classified examples and a few rounds of corrective feedback inside one conversation, learns the envelope of variation for a grammatical construction well enough to annotate unseen corpus sentences at better than 90% accuracy. The concrete task is binary: does a given token of consider instantiate the evaluative construction, as in 'Cain is considered a master of crime fiction,' or a cognition use, as in 'please consider all classes' or 'he considers running'? The paper reports 93% accuracy on the held-out evaluation set of 102 sentences, with recall of 94.9% and a Matthews correlation coefficient of 0.86. This is offered as evidence that the bottleneck of cleaning millions of corpus hits, where only 11.5% of a random sample of CONSIDER tokens were true positives, can be automated.","pith_inferences":["Editorial inference: because only 11.5% of raw CONSIDER tokens in the corpus are true positives, a classifier that simply said 'no' would already score about 88.5% accuracy, so the reported accuracy needs to be read together with recall and MCC, which the paper provides.","Editorial inference: a direct comparison of the pipeline against two independent human annotators on the same test sentences, measuring agreement rather than only model accuracy, would clarify how much of the residual error is annotation noise rather than model failure.","Editorial inference: the conversation-scoped training design means the method is most reliable for well-delimited formal distinctions, and extending it to semantically or pragmatically layered phenomena would require testing whether feedback generalizes across ambiguous edge cases, as the paper itself flags.","Editorial inference: a natural testable extension is to measure whether the same prompt transcript performs comparably on a different large language model, which the authors partially did with a newer model version."],"forward_implications":["A linguistic construction with millions of corpus hits can be annotated in full rather than sampled, so frequency and variant analyses can be run on complete populations instead of small cleaned subsets.","The method's roughly 60-minute training-to-evaluation cycle makes automated annotation affordable for researchers without programming skills, since the interaction happens through a chat interface.","The same pipeline should transfer to other clearly defined morphosyntactic variables, provided training data match the target corpus in register, time period, and variety.","With the model able to classify evaluative consider at scale, the substantive questions, whether the bare, as, and to be variants differ in meaning or are changing over time, become empirically tractable.","Evaluation metrics such as the Matthews correlation coefficient are the appropriate guardrail for imbalanced annotation tasks, and the authors recommend reporting the full confusion matrix."],"supporting_citations":[{"why":"Supplies the prompt-engineering design principles (clarity, examples, XML tags, chain-of-thought) that structure the pipeline.","marker":"Anthropic n.d."},{"why":"Provides the evidence that instructing the model to think step by step improves reasoning and classification performance.","marker":"Wei et al. 2022"},{"why":"Source of the NOW corpus data used for the pretraining and training sentences in the case study.","marker":"Davies 2016"},{"why":"Documents Sketch Engine's TenTen corpora, from which the 18-million-token EnTenTen21 search results are drawn.","marker":"Kilgariff et al. 2014"},{"why":"Justifies the Matthews correlation coefficient as the evaluation metric for imbalanced binary classification.","marker":"Chicco and Jurman 2020"},{"why":"Frames the LLM as a 'copilot for linguists,' the role this pipeline seeks to operationalize.","marker":"Torrent et al. 2024"},{"why":"Identifies the specific LLM whose conversation interface and in-context learning the method relies on.","marker":"Anthropic 2024a"}],"fun_headline_variants":["LLM learns 'consider' from 500 examples, hits 93% accuracy","93% accuracy: LLM copilot automates 'consider' annotation with tiny data","One-hour LLM loop labels consider tokens at 93% accuracy","LLM learns 'consider' grammar from 500 examples, beats 90% on test","AI copilot annotates 'consider' constructions at 93% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a 102-sentence held-out set, labelled once by the authors, accurately represents the variety and class distribution of the 18-million-token corpus, and that those labels are reliable.","fun_headline_variants_meta":{"raw":{"variants":["LLM learns 'consider' from 500 examples, hits 93% accuracy","93% accuracy: LLM copilot automates 'consider' annotation with tiny data","One-hour LLM loop labels consider tokens at 93% accuracy","LLM learns 'consider' grammar from 500 examples, beats 90% on test","AI copilot annotates 'consider' constructions at 93% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000731,"raw_usage":{"total_tokens":3261,"prompt_tokens":923,"completion_tokens":2338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2234}},"tokens_in":539,"tokens_out":2338,"duration_ms":17229,"temperature":1.0,"reasoning_tokens":2234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:43:49.726119+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the published prompt transcript and training data on a fresh stratified sample of, say, 1,000 CONSIDER sentences drawn from EnTenTen21, with each sentence labelled independently by two human annotators and the majority-class baseline computed; if model accuracy on true positives does not clearly exceed the roughly 88.5% baseline, or if human disagreement is comparable to the model's error rate, the claim that the method automates reliable annotation at scale would be falsified.","supporting_citations":[],"review_version":1}