REVIEW 2 major objections 5 minor 40 references
The paper claims that an automated hybrid parser-based detector can reliably measure children's filler-gap input and production at corpus scale, revealing construction-specific extraction biases and a generalization asymmetry in language mo
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 19:26 UTC pith:DVOHNGZK
load-bearing objection A genuinely useful automated detector and corpus-scale measurements, but the child-production numbers need a recall check on child speech before they can be treated as stable. the 2 major comments →
What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependencies
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that the detector is a viable solution for large-scale, fine-grained detection of target filler-gap structures even though it is imperfect. After validation against human annotations, the paper applies it to 2.84 million child-directed and child utterances from 57 CHILDES corpora and reports that matrix questions are an order of magnitude more frequent than embedded questions or relative clauses; object extraction outnumbers subject extraction in both question types; and relative clauses are roughly balanced between subject and object. The same fine-grained labels support a training intervention: removing matrix questions from a child-directed corpus significantly degrad
What carries the argument
The load-bearing tool is a hybrid detection pipeline that combines two parse types: constituency parses supply local structural signatures such as NP -> NP SBAR for relative clauses and SBARQ -> WHNP SQ for matrix questions, while dependency parses validate head-dependent relations such as relcl, nsubj, and dobj to pin down the extraction site. Additional heuristics, including a lexical filter over matrix verbs, separate embedded questions from free relatives. The detector outputs per-utterance labels for three constructions and subtypes (subject, object, adjunct, polar, possessive, reduced), which is the granularity that makes the corpus statistics and the filtered-training experiment possi
Load-bearing premise
The load-bearing premise is that errors made by the automatic parsers on noisy child speech do not systematically shift the counts of constructions or extraction sites; if parser failures are correlated with, say, fragmentary child questions, the reported object-bias and relative-clause balance could be artifacts rather than properties of the input.
What would settle it
Take a fresh sample of child-directed and child utterances from CHILDES, have linguists annotate every utterance for the three constructions and gap sites, then run the detector on the same utterances. If precision or recall on any category—especially adjunct relatives and reduced object relatives, where the paper's own validation is weakest—falls below 0.7, the corpus statistics would not support the conclusions.
If this is right
- Large-scale, fine-grained annotation of filler-gap constructions no longer requires manual trace annotation; the released detector and labels can be reused on any English CHILDES-style corpus.
- Children's filler-gap input is dominated by matrix questions by roughly an order of magnitude, so any frequency-driven account of acquisition must operate with construction- and extraction-site-specific counts rather than collapsed totals.
- Object-biased extraction is a property of wh-questions but not relative clauses in both adult input and child output, making extraction-site asymmetry construction-specific.
- In language models trained on child-directed data, matrix questions act as a source of generalization to embedded questions and relative clauses, but the rarer constructions do not feed back into matrix questions.
- The generated labels support future input-attribution studies that quantify which construction-specific training examples most affect model predictions.
Where Pith is reading between the lines
- A natural reading of the findings is that the input contains enough distributional evidence for a learner to induce extraction-site biases from frequency, weakening the poverty-of-stimulus argument for these constructions; the paper itself stops short of drawing that conclusion.
- The matrix-question transfer result suggests a finer-grained, testable prediction: removing object-extracted matrix questions should impair object-extracted embedded questions more than subject-extracted ones, since the shared dependency structure is the object gap.
- Because child production rankings track adult rankings in these cross-sectional snapshots, the paper establishes correlation, not causation; a longitudinal cross-lagged analysis on the same corpora could test whether adults' construction-specific rates actually predict later child rates.
- Extending the detector to clefting, topicalization, and free relatives would reveal whether the question/relative-clause split in extraction-site bias is a feature of filler-gap dependencies generally or specific to the three constructions studied.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper develops an automated annotation pipeline that combines constituency and dependency parsing to detect matrix wh-questions, embedded wh-questions, and relative clauses in English, with subtype labels for extraction site (subject, object, adjunct). The detector is evaluated in two ways: a manual evaluation of detector-positive CHILDES sentences (precision-oriented) and a comparison against Pearl and Sprouse's hand-annotated corpus (precision/recall on adult child-directed speech). The tool is then applied to 57 CHILDES corpora to estimate adult-input and child-production rates of filler-gap constructions over development, yielding claims about construction frequency, extraction-site asymmetries, and child-adult mirroring. A BabyLM filtered-corpus-training experiment tests whether removing one construction family from training data degrades performance on the same or other construction families.
Significance. If the detector is sufficiently accurate, this is a useful methodological contribution: it would be the first large-scale, fine-grained automated annotation of filler-gap constructions in CHILDES, enabling descriptive acquisition statistics and controlled LM training interventions that were previously impractical. The strengths of the paper include the release of code and data, validation against an external human-annotated corpus, the hybrid use of two parser types, and the controlled BabyLM ablation with random-sentence controls. The corpus-level statistics and the asymmetry results are also directly relevant to the poverty-of-the-stimulus debate. The main unresolved threat is the unmeasured recall of the detector on child production, which is load-bearing for the child-production trajectory claims.
major comments (2)
- [§4.1, §5 (Figs. 4, 6, 7)] The manual evaluation in §4.1 samples only sentences that the detector labeled as belonging to a target category; it therefore estimates precision, not recall, for child speech. The Pearl–Sprouse comparison in §4.2 provides recall estimates only for adult child-directed speech. Yet Figures 4 and 6 report child production rates and subject/object log-ratios, and §5.1 interprets child/adult mirroring. If fragmentary or ungrammatical child utterances fail at the parse stage before the construction heuristics apply, false negatives could be construction- or gap-site-specific and could create the observed trajectories and asymmetries. Please add a recall estimate on child speech—for example, by hand-annotating a random sample of child utterances (or a held-out set of child transcripts) and computing per-category recall—or explicitly restrict the child-production claims to precision-only state
- [Appendix C, §5, §6] The embedded-question detector depends on a hand-curated list of 17 matrix verbs derived from the top 50 verb lemmas in BabyLM. The paper does not report how many CHILDES embedded questions are headed by verbs outside this list, or a CHILDES-specific recall check for SEQ/OEQ. Because the embedded-question rates in §5 and the embedded-question ablation in §6 assume reasonably complete detection, a coverage analysis would be needed—for example, listing the matrix verbs that actually occur in potential embedded-question structures in CHILDES and the proportion captured by the 17-verb list—to show that the filter does not systematically undercount embedded questions in the target corpus.
minor comments (5)
- [§4.1, Table 2] The text says 'six core categories' and 'sampled 100 sentences from both child-directed speech and child speech,' but Table 2 lists eight categories and appears to report 100 sentences per category. Please clarify the sampling design.
- [Table 2] The metric labeled 'accuracy' is actually precision given the sampling of detector positives. Rename it 'precision' or clarify the denominator.
- [§5.1, Figure 4] The text claims that 'adjunct and polar questions outnumber subject and object extractions,' but Figure 4 groups these under 'Other,' so the claim is not directly visible. Please provide a separate table or figure showing adjunct and polar rates individually.
- [§5.3, Figure 8, Table 6] The significance test underlying the 'n.s.' and '***' markers is not described: the paper does not state whether the pairwise comparisons use bootstrap CIs, a paired test, or something else, nor does it give the minimum-count threshold used for the age bins.
- [Figure 6] The caption says 'Sizes of dot shades indicate number of total utterances within each bin,' but there is no legend or scale for the dot shading. Please add one or describe the mapping in the caption.
Circularity Check
No significant circularity: detector is validated against external gold annotations and the LM generalization study uses independent synthetic evaluations.
full rationale
The paper's claimed derivation is not circular. The detection system in §3 is a set of hand-written heuristics over constituency and dependency parses; no parameter is fitted to the corpus statistics that the paper later reports. The viability claim in §4.2 is supported by two external validations: a manual precision sample (§4.1, Table 2) and a comparison against the independently human-annotated Pearl & Sprouse (2013) corpus (§4.2, Table 3), which provides precision and recall. The CHILDES statistics in §5 are descriptive measurements produced by applying this detector, not predictions of those measurements. The BabyLM ablation in §6 uses the detector only to filter training corpora; the evaluation is on 3,432+5,000+5,000 synthetically constructed minimal pairs whose grammaticality judgments are defined independently of the detector. The paper explicitly acknowledges its limitations ('Our detectors are imperfect... the datasets we have extracted may not be suitable for research questions that require perfect identification'), which is an honest measurement-error caveat rather than an admission of circularity. No load-bearing self-citation or imported uniqueness theorem appears; the cited prior work (e.g., Pearl & Sprouse 2013; Boguraev et al. 2025) provides independent evidence or is used for comparison. The only substantive risk—that detector errors (especially unmeasured recall on child speech) could bias the §5 child/adult comparisons—is an empirical artifact risk, not a definitional reduction of the paper's outputs to its inputs.
Axiom & Free-Parameter Ledger
free parameters (4)
- Embedded-question matrix verb list =
[know, see, tell, look, remember, wonder, guess, ask, say, forget, figure, understand, decide, show, watch, hear, think]
- Age range and bin widths =
3-80 months; 6-month bins for Figure 4, 3-month bins for Figure 6; Laura 15-42 months
- Minimum-count threshold for subject-share comparisons =
not specified
- Epsilon in log ratio =
unspecified
axioms (8)
- domain assumption CHILDES transcripts are a representative sample of children's linguistic input and output.
- domain assumption Speaker metadata in CHILDES allows reliable separation of adult vs. child speech after age corrections.
- domain assumption spaCy and Berkeley Neural parser outputs are accurate enough for the heuristic detection rules.
- domain assumption Pearl & Sprouse (2013) trace annotations, converted by Appendix D heuristics, are valid gold labels.
- domain assumption Filler-gap theory (movement/traces) and extraction-site inference from surface configurations are the correct linguistic model.
- ad hoc to paper The hand-curated list of 17 question-embedding verbs is sufficient to distinguish embedded questions from free relatives in CHILDES/BabyLM.
- domain assumption Continuation probability on synthetic minimal pairs is a valid measure of FGD knowledge in language models.
- domain assumption Filtered-corpus control conditions (same token count, random sentence removal) isolate the effect of construction absence.
Cite this review
Pith. "Pith review of What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependencies." pith.science (2026). https://pith.science/paper/DVOHNGZK
@misc{pith2026260302082,
author = {Pith},
title = {Pith review of: What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependencies},
year = {2026},
howpublished = {\url{https://pith.science/paper/DVOHNGZK}},
note = {Machine review of arXiv:2603.02082}
}
read the original abstract
Children's acquisition of filler-gap dependencies has been argued by some to depend on innate grammatical knowledge, while others suggest that the distributional evidence available in child-directed speech suffices. Unfortunately, the relevant input is difficult to quantify at scale with fine granularity, making this question difficult to resolve. We present a system that identifies three core filler-gap constructions in spoken English corpora -- matrix wh-questions, embedded wh-questions, and relative clauses -- and further identifies the extraction site (i.e., subject vs. object vs. adjunct). Our approach combines constituency and dependency parsing, leveraging their complementary strengths for construction classification and extraction site identification. We validate the system on human-annotated data and find that it scores well across most categories. Applying the system to 57 English CHILDES corpora, we are able to characterize children's filler-gap input and their filler-gap production trajectories over the course of development, including construction-specific frequencies and extraction-site asymmetries. The resulting fine-grained labels enable future work in both acquisition and computational studies, which we demonstrate with a case study using filtered corpus training with language models.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Ben Ambridge, Evan Kidd, Caroline F Rowland, and Anna L Theakston. 2015. The ubiquity of frequency effects in first language acquisition. Journal of Child Language, 42(2):239--273
2015
-
[4]
Emily Atkinson, Matthew W Wagers, Jeffrey Lidz, Colin Phillips, and Akira Omaki. 2018. Developing incrementality in filler-gap dependency processing. Cognition, 179:132--149
2018
-
[5]
Lois Bloom, Susan Merkin, and Janet Wootten. 1982. ``wh''- Questions : Linguistic Factors That Contribute to the Sequence of Acquisition . Child development, pages 1084--1092
1982
-
[6]
Sasha Boguraev, Christopher Potts, and Kyle Mahowald. 2025. Causal Interventions Reveal Shared Structure Across English Filler -- Gap Constructions . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25032--25053
2025
-
[7]
Susan R Braunwald. 1971. Mother-child communication: The function of maternal-language input. Word, 27(1-3):28--50
1971
-
[8]
Roger Brown. 1973. A first language: The early stages. Harvard University Press, Cambridge, MA
1973
-
[9]
Iona Carslaw, Sivan Milton, Nicolas Navarre, Ciyang Qing, and Wataru Uegaki. 2025. Automatic extraction of clausal embedding based on large-scale E nglish text data. In Proceedings of the Society for Computation in Linguistics 2025, pages 322--332
2025
-
[10]
Chi-Yun Chang, Xueyang Huang, Humaira Nasir, Shane Storks, Olawale Akingbade, and Huteng Dai. 2025. Mind the Gap : How BabyLMs Learn Filler - Gap Dependencies . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15060--15076
2025
-
[11]
Noam Chomsky. 1977. On W h-movement. In Formal Syntax , pages 71--132. Academic Press, New York
1977
-
[12]
Noam Chomsky. 1986. Knowledge of Language : Its Nature , Origin , and Use . Convergence. Praeger, New York
1986
-
[13]
Atrey Desai and Sathvik Nair. 2026. https://arxiv.org/abs/2604.14459 Filling in the Mechanisms : How do LMs Learn Filler - Gap Dependencies under Developmental Constraints ? Preprint, arXiv:2604.14459
Pith/arXiv arXiv 2026
-
[14]
Holger Diessel and Michael Tomasello. 2005. A New Look at the Acquisition of Relative Clauses . Language, 81(1):1--25
2005
-
[15]
Michael C Frank. 2023. Bridging the data gap between children and large language models. Trends in Cognitive Sciences, 27(11):990--992
2023
-
[16]
Annie Gagliardi, Tara M Mease, and Jeffrey Lidz. 2016. Discontinuous development in the acquisition of filler-gap dependencies: Evidence from 15-and 20-month-olds. Language Acquisition, 23(3):234--260
2016
-
[17]
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. 2023. https://arxiv.org/abs/2308.03296 Studying Large Language Model Generalization with Influence Func...
Pith/arXiv arXiv 2023
-
[18]
Yiding Hao. 2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.28 Evaluating Attribution Methods using White - Box LSTM s . In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 300--313, Online. Association for Computational Linguistics
-
[19]
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. https://doi.org/10.5281/zenodo.1212303 spaCy: Industrial-strength Natural Language Processing in Python
-
[20]
Katherine Howitt, Sathvik Nair, Allison Dods, and Robert Melvin Hopkins. 2024. Generalizations across filler-gap dependencies in neural language models. In Proceedings of the 28th conference on computational natural language learning, pages 269--279
2024
-
[21]
Yaling Hsiao, Nicola J Dawson, Nilanjana Banerji, and Kate Nation. 2023. The nature and frequency of relative clauses in the language children hear and the language children read: A developmental cross-corpus analysis of English complex grammar. Journal of Child Language, 50(3):555--580
2023
-
[22]
Rodney D Huddleston, Geoffrey K Pullum, and Laurie Bauer. 2002. The Cambridge grammar of the English language . Cambridge University Press Cambridge
2002
-
[23]
Jaap Jumelet, Milica Denic, Jakub Szymanik, Dieuwke Hupkes, and Shane Steinert-Threlkeld. 2021. Language models use monotonicity to assess NPI licensing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4958--4969
2021
-
[24]
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. https://doi.org/10.18653/v1/P19-1340 Multilingual constituency parsing with self-attention and pre-training . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3499--3505, Florence, Italy. Association for Computational Linguistics
-
[25]
Nikita Kitaev and Dan Klein. 2018. https://doi.org/10.18653/v1/P18-1249 Constituency parsing with a self-attentive encoder . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676--2686, Melbourne, Australia. Association for Computational Linguistics
-
[26]
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885--1894. PMLR
2017
-
[27]
Nur Lan, Emmanuel Chemla, and Roni Katzir. 2024. Large language models and the argument from the poverty of the stimulus. Linguistic Inquiry, pages 1--28
2024
-
[28]
Julie Anne Legate and Charles D Yang. 2002. Empirical re-assessment of stimulus poverty arguments. The Linguistic Review, 19(1-2):151--162
2002
-
[29]
Brian MacWhinney. 2000. The CHILDES Project : Tools for Analyzing Talk . V olume II : The Database , 3rd edition. Lawrence Erlbaum Associates, Mahwah, NJ
2000
-
[30]
Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing AANN s. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913--929
2024
-
[31]
Abhinav Patil, Jaap Jumelet, Yu Ying Chiu, Andy Lapastora, Peter Shen, Lexie Wang, Clevis Willrich, and Shane Steinert-Threlkeld. 2024. Filtered corpus training ( FiCT ) shows that language models can generalize from indirect evidence. Transactions of the Association for Computational Linguistics, 12:1597--1615
2024
-
[32]
Lisa Pearl and Jon Sprouse. 2013. Computational models of acquisition for islands. Experimental Syntax and Islands Effects, pages 109--131
2013
-
[33]
Geoffrey K Pullum and Barbara C Scholz. 2002. Empirical assessment of stimulus poverty arguments. The Linguistic Review, 19(1-2):9--50
2002
-
[34]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9
2019
-
[35]
Caroline F Rowland, Julian M Pine, Elena VM Lieven, and Anna L Theakston. 2003. Determinants of acquisition order in wh-questions: Re-evaluating the role of caregiver speech. Journal of Child Language, 30(3):609--635
2003
-
[36]
Alessandro Sanchez, Stephan C Meylan, Mika Braginsky, Kyle E MacDonald, Daniel Yurovsky, and Michael C Frank. 2019. childes-db: A flexible and reproducible interface to the child language data exchange system. Behavior Research Methods, 51(4):1928--1941
2019
-
[37]
Jon Sprouse, Ivano Caponigro, Ciro Greco, and Carlo Cecchetto. 2016. Experimental syntax and the variation of island effects in English and Italian . Natural Language & Linguistic Theory, 34(1):307--344
2016
-
[38]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 LLaMA : Open and Efficient Foundation Language Models . Preprint, arXiv:2302.13971
Pith/arXiv arXiv 2023
-
[39]
Virginia Valian. 1991. https://doi.org/10.1016/0010-0277(91)90046-7 Syntactic subjects in the early speech of American and Italian children . Cognition, 40(1-2):21--81
-
[40]
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023. Findings of the BabyLM challenge: Sample -efficient pretraining on developmentally plausible corpora. In Proceedings of the BabyLM Challenge at the 27th Conference on Computationa...
2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.