Computation and Language · arXiv:2609.37635 v1 · submitted 29 Sep 2026 · 7 references
Co-Linguistics: AI-augmented Theory Construction in Linguistics
The paper says
“LLMs have been studied in recent linguistics as potential models of humans' linguistic abilities.”
more of the abstract
Here we discuss an entirely different use of AI, namely as a co-scientist, to help construct and assess linguistic theories (we refer to the result as "Co-Linguistics"). Since the 1960s, linguistics has developed theories that are in principle mathematically formalizable, often in the language of formal language theory or model theory. The AI revolution in mathematics will thus have consequences in linguistics-but with an essential twist: proving new theorems is rarely the linguist's goal. Rather, one seeks to find the best set of axioms to derive empirical statements. AI could accelerate research by making existing theories fully explicit, by comparing competing theories, and more ambitiously, by proposing new theories (in machine learning, this relates to "program induction"). It will also help assess theories by accelerating the identification and test of crucial predictions, thanks to unparalleled access to data (in machine learning, this relates to "active learning"). While the cycle from theory evaluation to theory construction may give rise to recursive and possibly autonomous improvement of linguistic theories, humans remain central: linguists provide scientific directions and evaluate theories conceptually, and experimental participants are needed to assess empirical predictions that are outside the reach of LLMs.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 30 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. An honest, well-hedged position paper whose case studies are entirely self-reported and unshipped; the architecture is borrowed but the linguistics-specific framing is a real contribution, and it deserves a serious referee. This line is wrong
3 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 7 references say.
The paper says, at §2.1 and Appendix (b49, b53, b57, b61)
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.76):
“Correspondingly, our results in the Appendix are provisional: more work is needed to check that the AI formalization matches what the original theories intended. 2.2 Comparing existing theories Once theories are formalized, one can seek to prove equivalence or non-equivalence results among them, and to isolate crucial predictions that may adjudicate among them.”
Pith says: Objection 1 of 3 · major
The central feasibility claim (b117) currently rests on reported case studies whose verification artifacts are not in the manuscript. The text states that Claude offered restatements and Lean verifications (b49), that predictions were re-proved and extended (b53), and that connection theorems were obtained (b61), but no Lean file, theorem statement, proof transcript, or reproducible interaction log is provided, and the Appendix is a mutable online document reached through a tinyurl link.
more of objection 1
The authors' honest provisionality caveat (b57) does not make these claims checkable. I request that at least one complete, stable case study—definitions, theorem statements, and machine-checked proofs or reproducible code—be included in the paper or in a versioned companion artifact, so that the formalization can be independently checked.
The paper says, at §2.3 (b67-b69)
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The claim that the Codex-assisted revision of Spector's anaphora theory "has better empirical coverage" is a scientific adequacy claim, not a formalization-fidelity claim, and it is not supported by any derivation or test shown in the manuscript. To make the claim, the paper should demonstrate that the revised axioms still derive Spector's positive predictions for left-to-right licensing, derive the cataphoric pattern ("Either it_i is upstairs, or there isn't a bathroom_i"), and do not license unavailable configurations.
more of objection 2
If this is intended only as an illustration of a workflow rather than an established result, that should be stated explicitly and the adequacy claim withdrawn or deferred.
The paper says, at §2.2 (b61)
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The comparison of Limited Symmetry with Transparency/Local Contexts, and the checking and correction of the five-theory equivalence results, rely on an "unpublished addition to Schlenker (2009)" and on AI-generated general theorems; neither the hand-written proofs nor the AI-generated statements are available to the reader.
more of objection 3
Without a stable statement of at least the main equivalence and non-equivalence theorems and of the corrections introduced, this case study cannot serve as evidence that AI accelerates theory comparison. At minimum, include a written statement of the theorems and indicate the corrections in a versioned appendix.
4 minor objections
The paper says, at §3.2 (b114)
Pith says: minor objection 1 of 4
"Human linguistics abilities" should read "human linguistic abilities".
The paper says, at §1.1
Pith says: minor objection 2 of 4
The terms "program induction" and "active learning" are invoked informally; a one-sentence definition at first use would help readers outside machine learning.
show 2 more
The paper says, at §2.3 (b69)
Pith says: minor objection 3 of 4
The phrase "seems no more complex" would benefit from a stated measure of complexity; otherwise the comparison with Spector's original theory is difficult to evaluate.
The paper says, at §3.1, item 1
Pith says: minor objection 4 of 4
The operationalization of parsimony by code length is acknowledged as only an initial approximation, but the section should explicitly say that length is relative to an implementation language and is not a direct measure of theory complexity, so that cross-language comparisons are not overinterpreted.
What would settle it
Independently reproduce the flagship formalization case from the published record alone, without consulting the authors' appendix or any prior LLM output: formalize the geometric classifier semantics of Schlenker & Lamberton (2024) in Lean 4 starting only from the published paper. Then check (i) whether the two 'equivalent restatements' claimed in §2.1 (b49) compile in Lean and are mechanically proved equivalent to this independent formalization, and (ii) whether they are in fact simpler under an explicit measure such as Lean term length. If independent formalization converges on the same statements and the equivalences compile, the formalization-and-verification pillar of b117 is supported.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 30 Sep 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Read it if you want a compact statement of the Co-Linguistics program. It is a position paper, not a result paper, and the authors are upfront about that. The central architecture (LLM coupled with a proof assistant, driving a recursive theory-improvement loop) is not theirs; they credit Chojecki, Chatzikyriakidis and Luo, and Ellis et al. themselves. What is genuinely new is the application to linguistics, and specifically formal semantics, where the goal is finding the right axioms to derive empirical statements rather than proving theorems. That framing is the paper's best contribution: it correctly identifies why the math-AI revolution transfers to linguistics with a twist. The paper does several things well. It is candid about the human-in-the-loop necessity (b8, b27), about the fidelity problem -- checking that the formalization matches the intended theory is the linguist's job (b55-b57) -- and about LLM proxies being error-prone (b94). No overclaiming, no p-hacking, no circular derivation. The discussion contrasting Bayesian program induction (Ellis et al.) with the unconstrained LLM search is genuinely clarifying. The soft spots are real but proportionate. First, the entire evidential base is self-reported: the Lean-verified restatements (b49), the Limited Symmetry re-derivations and connection theorems (b53, b61), all deferred to a mutable tinyurl appendix with no artifacts in the paper itself. Nothing is checkable from the text. The authors flag this as provisional (b57), which is honest, but honesty does not close the gap. Second, the sharpest problem: the claim that the revised anaphora theory "has better empirical coverage" (b69) is a scientific adequacy claim, not a formalization-fidelity claim, so the hedge does not cover it. Supporting it requires showing the revised axioms preserve Spector's positive predictions while adding the disjunctive cataphora; no derivation or test is shown. Third, the five-theory equivalence results rest on an unpublished addition to Schlenker (2009); building the flagship case on an in-circle, un-auditable source is a structural weakness, and the paper should name it. Who it is for: anyone working on AI-assisted formal work in linguistics, and editors deciding whether the field should have this conversation in print. It deserves a serious referee and is publishable as a position statement, but the case studies need to ship as stable, accessible artifacts -- Lean files, code, the formalized theories -- before the feasibility claim is treated as established.
The strongest objection
The central claim, b117 ('AI could prove to be a powerful tool to formalize, verify, and even construct theories in linguistics'), is a feasibility claim whose only evidence is three case studies, all deferred to a mutable online appendix reached through a tinyurl link, with no artifacts in the paper itself: the Lean-verified classifier restatements (b49), the Limited Symmetry re-derivations and new quantifier facts (b53), the connection theorems (b61), and the revised anaphora theory (b68-b69). Three independent verification layers are each outstanding. First, formal correctness: the paper states that Claude 'offered verifications of the equivalences using the proof-assistant Lean' (b49), but no Lean file, theorem statement, or proof appears in the text, so the soundness of even layer one is a report, not a demonstrated result. Second, fidelity: the authors themselves mark this provisional (b55-b57); that is honest, but it does not close the gap. Third, and most consequential, the claim that the new anaphora theory 'has better empirical coverage' (b69) is a scientific adequacy claim about the fragment (e.g., cataphora in disjunction, 'Either it_i is upstairs, or there isn't a bathroom_i'), and it is not covered by the formalization-fidelity hedge at all: supporting it would require showing the revised axioms preserve Spector's positive predictions while adding the disjunctive ones, and no derivation or test is shown. A compounding issue is that the five-theory equivalence results the LLM 'checked and corrected' rest on a hand-written 'unpublished addition to Schlenker (2009)', a base the community cannot audit because it is not a published source. The paper is appropriately hedged and I find no dishonesty or overclaim, but the entire evidential base is a self-report, so the feasibility claim outruns the demonstrated evidence.
The reader's weakest assumption — fidelity of AI-generated formalizations to the linguist's intended theory, evidenced only by unshipped anecdotes — is exactly the load-bearing concern I identify. I extend it only slightly: alongside the fidelity layer the paper itself flags as provisional (b55-b57), there is a formal-correctness layer (no Lean artifacts in the paper for the claimed verifications of b49 and b53) and a distinct empirical-adequacy layer (the 'better empirical coverage' claim of b69), which the fidelity hedge does not even attempt to cover. The paper is a vision piece, honestly hedged, with no red flags of circular derivation or overclaim; its contribution is the co-linguistics architecture and its placement within ongoing AI-for-science work. The distinctive, load-bearing evidence — that the loop actually works on real semantic theories — is self-reported and unimpeachable through the text. That justifies the reader's CONDITIONAL verdict: conditional on the authors shipping stable, auditable artifacts (Lean files, theorem statements, and the axiom sets and prediction derivations for the anaphora revision) so that a third party can reproduce the formalization and the claimed empirical coverage. I do not think the evidence supports moving the verdict to REJECT; a position paper may legitimately publish a programmatic proposal with provisional evidence, provided the condition is stated, which the paper itself does. Conversely, ACCEPT would be premature, because the paper's own prose (b57) concedes that the central promised capability is not yet demonstrated. UNCHANGED is therefore the appropriate verdict: the reader's conditional judgment is the right one, and the concrete test proposed above is the minimal check that would resolve whether the condition has been met.
The referee's report
If the proposed workflow is viable, it gives a concrete route toward realizing the long-standing program of fully formal, axiomatic linguistic theories, with machine-checked derivations and systematic comparison of alternative axiom sets. The paper connects linguistics to current AI-for-mathematics efforts, Bayesian program induction, and active learning, and it usefully distinguishes the mathematical goal of proving theorems from the linguistic goal of selecting axioms that derive empirical statements. The authors deserve credit for an appropriately hedged framing: they state that AI formalizations need human checking (b55-b56), that Appendix results are provisional (b57), that LLM proxies for human judgments are error-prone (b94), and that human judgments are ultimately required (b98). However, the feasibility case is currently supported only by reported anecdotes whose artifacts are not present in the manuscript, and the strongest constructive claim—the improved anaphora theory—involves an empirical adequacy judgment that is not demonstrated.
Pith's reading of the paper (a machine's summary, not the authors')
The paper proposes a new role for large language models in linguistics: as a co-scientist that helps build and assess theories, rather than as a model of human linguistic ability. It argues that since formal linguistic theories are in principle collections of axioms that mechanically derive empirical statements, the AI revolution in mathematics should carry over to linguistics—but with a twist: proving theorems is rarely the goal, so the real task is finding the best set of axioms to derive desirable empirical claims. The authors report proof-of-concept case studies in which an LLM restated a semantic theory in cleaner geometric terms with Lean-verified equivalences, derived new predictions from an existing presupposition theory, and revised an anaphora theory to fix a known empirical gap. They argue that when formalization, comparison, construction, and assessment are all AI-assisted, theories may improve recursively, with humans remaining the conceptual judges and the collectors of data that lie outside an LLM's reach.
What carries the argument. The operative mechanism is a recursive four-step loop: AI constructs candidate theories, checks that conclusions follow from principles using a proof assistant such as Lean, assesses competing theories by parsimony (operationalized by code length as a proxy for Minimum Description Length) and by empirical coverage, and finds new data to adjudicate among them. The trust architecture couples untrusted LLM generators with a trusted proof assistant, so that the machine may propose freely but every step is mechanically checked. The machine-learning analogues the paper names are program induction for theory construction and active learning for prediction testing.
Core claim. The central claim is that AI, coupled with proof assistants, can accelerate the whole cycle of linguistic theory work: making existing theories fully explicit, proving equivalences and non-equivalences among competing theories, constructing new ones, and identifying crucial predictions to test. The distinctive twist relative to AI for mathematics is that the difficulty rarely lies in proving a derivation; it lies in choosing the axioms. On that assumption, the paper reports three results: two equivalent, simpler geometric restatements of the classifier semantics of Schlenker and Lamberton (2024), verified with Lean; derivation of predictions of Kalomoiros's Limited Symmetry theory of presupposition projection for quantifiers not previously discussed; and a revision of Spector's (2026) anaphora theory, built on Kalomoiros's framework, that solves the cataphora problem while preserving the original theory's positive predictions.
The premise it rests on. The whole program rests on the assumption that an AI-generated formalization faithfully captures the theory the linguist actually has in mind, a fidelity the paper itself flags as provisional and not yet demonstrated by artifacts in the paper itself.
If the paper is right (Pith's inferences, not the authors')
Linguists could outsource much of the formalization and proof-checking of existing theories, converting informal claims into Lean-verified statements.
Competing formal theories could be mechanically compared for equivalence and non-equivalence, and their crucial points of divergence isolated.
AI could propose genuinely new theories: the paper's revised anaphora theory is claimed to match the complexity of the original while extending its empirical coverage.
Theory evaluation could be partly automated through parsimony measures and large-scale literature review, accelerating the cycle from assessment to construction.
When the object of study is the linguistic behavior of LLMs themselves—and increasingly for animal communication as corpus sizes grow—the empirical part of the loop may become fully automated.
Editorial inference: the same untrusted-generator/trusted-checker loop should transfer to any field whose theories are formalizable collections of axioms, so the proposal is a template rather than a linguistics-specific tool.
Editorial inference: the paper's division of labor implies that the scientific bottleneck shifts from proving theorems to designing discriminating experiments at the margins of human judgment, where LLM proxies are least reliable.
Editorial inference: a testable extension would measure whether the AI-proposed anaphora theory predicts acceptability or inference patterns that were not part of its construction, rather than merely re-deriving published judgments.
Editorial inference: if the program works, the value of a linguist shifts toward curating axiom sets and adjudicating conceptual adequacy, since the mechanical proof steps are increasingly machine-checkable.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
LLMs have been studied in recent linguistics as potential models of humans' linguistic abilities. Here we discuss an entirely different use of AI, namely as a co-scientist, to help construct and assess linguistic theories (we refer to the result as "Co-Linguistics"). Since the 1960s, linguistics has developed theories that are in principle mathematically formalizable, often in the language of formal language theory or model theory. The AI revolution in mathematics will thus have consequences in linguistics-but with an essential twist: proving new theorems is rarely the linguist's goal. Rather, one seeks to find the best set of axioms to derive empirical statements. AI could accelerate research by making existing theories fully explicit, by comparing competing theories, and more ambitiously, by proposing new theories (in machine learning, this relates to "program induction"). It will also help assess theories by accelerating the identification and test of crucial predictions, thanks to unparalleled access to data (in machine learning, this relates to "active learning"). While the cycle from theory evaluation to theory construction may give rise to recursive and possibly autonomous improvement of linguistic theories, humans remain central: linguists provide scientific directions and evaluate theories conceptually, and experimental participants are needed to assess empirical predictions that are outside the reach of LLMs.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-30
The authors' main claim, as the classifier picked it out 0.62
“While the cycle from theory evaluation to theory construction may give rise to recursive and possibly autonomous improvement of linguistic theories, humans remain central: linguists provide scientific directions and evaluate theories conceptually, and experimental participants are needed to assess empirical predictions that are outside the reach of LLMs.”
How the main claim is stated: hedged possibility 0.73
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.78 · human participants 0.82 · controlled comparison 0.03 · randomized assignment 0.02 · states quantitative result 0.02 · compares with prior results 0.23 · mentions code or data availability 0.02
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| LLMs have been studied in recent linguistics as potential models of humans' linguistic abilities. | background 1.00 | 0.04 | 0.02 | asserted 0.90 |
| Here we discuss an entirely different use of AI, namely as a co-scientist, to help construct and assess linguistic theories (we refer to the result as "Co-Linguistics"). | contribution 1.00 | 0.39 | 0.27 | asserted 0.90 |
| Since the 1960s, linguistics has developed theories that are in principle mathematically formalizable, often in the language of formal language theory or model theory. | background 1.00 | 0.03 | 0.03 | asserted 1.00 |
| The AI revolution in mathematics will thus have consequences in linguistics-but with an essential twist: proving new theorems is rarely the linguist's goal. | background 0.87 | 0.05 | 0.07 | asserted 0.66 |
| Rather, one seeks to find the best set of axioms to derive empirical statements. | background 0.93 | 0.05 | 0.07 | asserted 1.00 |
| AI could accelerate research by making existing theories fully explicit, by comparing competing theories, and more ambitiously, by proposing new theories (in machine learning, this relates to "program induction"). | contribution 0.47 | 0.06 | 0.14 | hedged possibility 1.00 |
| It will also help assess theories by accelerating the identification and test of crucial predictions, thanks to unparalleled access to data (in machine learning, this relates to "active learning"). | contribution 0.44 | 0.07 | 0.08 | hedged possibility 0.73 |
| While the cycle from theory evaluation to theory construction may give rise to recursive and possibly autonomous improvement of linguistic theories, humans remain central: linguists provide scientific directions and evaluate theories conceptually, and experimental participants are needed to assess empirical predictions that are outside the reach of LLMs. | limitation 0.67 | 0.08 | 0.51 | hedged possibility 0.73 |
What kind of paper the classifier reports this is: position or commentary 1.00 · evidence from argument 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 6,340 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (7)
- [1] Bagaria, Joan, "Set Theory" (2023), The Stanford Encyclopedia of Philosophy (Spring 2023 Edition), Edward N. Zalta & Uri Nodelman (eds.), https://plato.stanford.edu/archives/spr2023/entries/set-theory/ Beguš, Gašper (2021). CiwGAN and fiwGAN: Encoding information in acoustic d... on PithPith's reviewarXiv
- [7] https://medium.com/@BerendWatchusIndependent/a-new-paradigm-for-scientific-discovery-from-indu ctive-learning-to-axiom-based-reasoning-6433bbe4b088 Wilcox, E., Levy, R., Morita, T., & Futrell, R. (2018). What do RNN language models learn about filler–gap dependencies? In T. Li... on PithPith's reviewarXiv
- [617] https://doi.org/10.1057/s41599-025-04907-8. Watchus, Berend: 2025, A New Paradigm for Scientific Discovery: From Inductive Learning to Axiom-Based Reasoning. Blog post, Medium, September 9, on Pithdoi:10.1057/s41599-025-04907-8
- [2024] “Can large language models help augment English psycholinguistic datasets?” Behavior Research Methods 56: 6082–6100. https://doi.org/10.3758/s13428-024-02337-z . Vázquez Martínez, Héctor Javier and Charles Yang (2026). Measuring Form and Function in Language Models. https://ar... on PithPith's reviewarXiv
- [2025] https://drive.google.com/file/d/1efjofSDjDeP1MfRgjtHEvmiudr20ZhZF/ Schlenker, Philippe; Chemla, Emmanuel; Schel, Anne; Fuller, James; Gautier, Jean-Pierre; Kuhn, Jeremy; Veselinovic, Dunja; Arnold, Kate; Cäsar, Cristiane; Keenan, Sumir; Lemasson, Alban; Ouattara, Karim; Ryder,... on PithPith's reviewarXiv
- [2026] “Can large language models be viewed as a cognitive model of human language? Not yet, regardless of reasoning capability and size.” Acta Psychologica 267: 107025. https://doi.org/10.1016/j.actpsy.2026.107025 . Kalomoiros, Alexandros (2022). Deriving the (a)-symmetries of presu... on PitharXiv
- [5024] Garg, Nikhjil (2026). EconCSLib: AI-Assisted Lean Formalization for Economics & Computation research. https://arxiv.org/abs/2606.13306 7 Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. (2020). SyntaxGym: An Online Platform for Targeted Evaluation of Languag... on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment