Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that humans can acquire a constructed language from a grammar book and dictionary and answer commonsense questions in it at 87% accuracy, while GPT-5, the best tested model, reaches 47%—and that human-verified reasoning tra

desk verdict A clever pilot with a real leakage-resistant idea, but the human baseline may be the person who built the test set—so the headline 'humans 87%, LLMs 47%' is not yet a finding. read the letter →

arxiv 2509.00425 v1 pith:XEUOOSZC submitted 2025-08-30 cs.CL

Fenghua Liu , Yulong Chen , Yixuan Liu , Zhujun Jin , Solomon Tsai , Ming Zhong This is my paper
classification cs.CL
keywords CamlangconstructedlanguagemetalinguisticreasoningLLMevaluationexplicitgrammarlearningCommonsenseQAexact-matchaccuracyhumanverification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show that current LLMs cannot do what human adults can: learn a genuinely new language from explicit rules alone. It builds Camlang, a linguistically plausible but unattested constructed language with a grammar book and English–Camlang dictionary, then translates 47 CommonsenseQA questions into it. A human participant scored 87% using only the provided resources; GPT-5 scored 47%, and all other tested reasoning models did worse. Human inspection of model answer traces showed almost none contained correct, complete parsing and semantic interpretation; most correct answers depended on matching words to dictionary entries and guessing from English commonsense priors. If the result holds, familiar-language benchmark scores overstate LLM metalinguistic competence, and novel constructed languages provide a sharper test.

What carries the argument

Camlang itself is the central object: a constructed language built from typologically attested features (vowel harmony, consonant mutation, head-marking morphology, verb fronting) combined in a configuration that the authors say is unattested. It comes with two resources—a grammar book specifying phonology and morphosyntax, and an English–Camlang dictionary—so that learning requires explicit rule application and lexical lookup. This separation does the diagnostic work: the grammar book lets errors be classified as parsing failures, the dictionary isolates lexical-semantic failures, and the novelty of the language rules out memorisation from pre-training data.

What would settle it

Give GPT-5 the same 47 questions with the same dictionary but a word-shuffled grammar book; if accuracy stays near 47%, the grammar rules are not carrying the successes, and if it drops substantially, they are. Paired with a second human group of at least ten blind learners, this would show whether the human-model gap is robust or an artifact of the single-participant, hand-curated test set.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a separation: Camlang-CSQA-v0 holds commonsense knowledge constant and varies only the linguistic system, so the bottleneck for LLMs shifts from knowing the world to learning a language from explicit rules. Models solve the English version at 85–98% exact match but fall to 21–47% on Camlang, while a human participant using only the grammar book and dictionary reaches 87%. Human verification of reasoning traces shows that almost none of the models' correct answers are backed by a correct and complete parse of the question together with correct question and option meaning; GPT-5's traces are more often incomplete than wrong, suggesting partial rule-fo

Load-bearing premise

The load-bearing premise is that the 47 hand-curated Camlang items—with entities swapped to fit the dictionary, options edited, and gold answers chosen by the annotators—are a valid, difficulty-neutral measure of metalinguistic reasoning, with the single human participant providing the 87% baseline.

Editorial extensions

If this is right

  • If the result holds, exact-match accuracy on familiar-language benchmarks substantially overstates LLM linguistic competence; human-verified accuracy is near zero for complete rule-following even when exact match is high.
  • The best tested model, GPT-5 in the context-only setting, remains about 40 points behind the human participant on Camlang, so state-of-the-art reasoning models cannot currently be relied on for explicit rule-and-dictionary language learning.
  • Providing LLMs with tool-based file search and code interpreter access did not close the gap and sometimes lowered accuracy, because surface forms must be lemmatised before dictionary lookup; resource delivery is not the bottleneck.
  • Because Camlang is constructed and its full resources are not released, it provides a leakage-controlled substrate that can be reused to build further tasks such as translation, parsing, and mathematical reasoning within the same benchmark suite.
  • A human participant can reach 87% with only the grammar book and dictionary, so poor model performance is not attributable to the artificiality of Camlang.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to split the items by how many answer options share a dictionary word with the question; if model accuracy concentrates on cue-rich items, that would confirm shallow alignment is the mechanism rather than a side effect of the verification labels.
  • The single-human 87% baseline is best read as existence evidence that Camlang is human-learnable, not as a stable competence estimate; a larger blind human sample could widen or narrow the gap, so the robust claim is relative (models below humans), not absolute.
  • Because tool-based lookup hurt accuracy, a targeted intervention—teaching the model to segment surface forms before querying the dictionary—could isolate whether the bottleneck is morphological analysis or resource use; the paper implies this but does not test it.
  • The design also suggests a cheap contamination probe: ask an LLM to complete Camlang sentences after seeing only half the grammar; if performance does not scale with grammar completeness, priors and memorisation are driving answers rather than rule application.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Camlang, a constructed language with an explicit grammar book and bilingual dictionary, and constructs Camlang-CSQA-v0, a 47-item multiple-choice QA test set adapted from CommonsenseQA. It evaluates several LLMs (GPT-4o, GPT-o3, GPT-o4-mini, GPT-5, DeepSeek-R1) in zero-shot settings, with both tool-enabled and context-only resource access, and compares them to one human participant. The main empirical claims are: (i) LLMs drop sharply from 85–98% accuracy on the English version to 21–47% on Camlang; (ii) the human participant reaches 87.23% on Camlang; and (iii) human verification of model reasoning traces shows that most model successes rely on shallow lexical alignment rather than systematic metalinguistic deduction. The paper is explicitly labelled 'Working in Progress' and states that results are preliminary.

Significance. The core idea is timely and potentially valuable: a novel, typologically plausible constructed language with separate grammar and dictionary resources could provide a contamination-resistant testbed for rule application and metalinguistic reasoning. The English control is a useful design choice, and the human-verification tier (SHV/MHV/LHV) is an interesting attempt to go beyond exact-match accuracy. However, the significance of the reported findings is currently limited by the human baseline being a single participant whose naivety is not established, by author-curated gold labels, and by the withholding of the grammar and dictionary. If the human comparison is confirmed with a proper naive-participant study, the paper would make a meaningful contribution; at present, the central comparative claim is not sufficiently supported.

major comments (4)
  1. [§4.2] The human baseline is load-bearing for the abstract's 'far below human performance at 87%' claim. The single participant is described only as 'a first-year undergraduate student in Linguistics at Oxford', and the text then says 'we ask the annotator to complete the English questions after the Camlang questions.' Given §4.3 refers to 'the core annotator, who participates in Camlang development', the manuscript never rules out that the human participant is the same person who constructed the test set and selected gold answers. If so, the 87.23% score reflects answer-key familiarity rather than deductive learning from grammar and dictionary. Even if the participant was naive, n=1 provides no variance estimate and cannot support a general claim about human metalinguistic competence. This needs multiple independent, screened, naive participants, with order counterbalancing and a clear stateme
  2. [§3.3] The gold labels of Camlang-CSQA-v0 are not taken from an existing benchmark: the authors randomly selected 50 CommonsenseQA items, replaced entities (e.g., 'weasel' to 'cat'), modified or replaced answer options, inserted a 'none of the above' option, and then 'manually select[ed] the gold answers.' This makes the test set a private, author-curated instrument. If the same annotator also served as the human baseline, the human-model gap could be inflated by shared construction conventions rather than by rule application. The manuscript should report exactly which items were modified, provide a second independent annotation of gold answers with agreement statistics, and ideally validate the items on a separate group of naive participants before using them to compare models and humans.
  3. [§5.3.1] The full grammar book and dictionary are withheld: 'we do not release the full grammar book and dictionary at this stage.' This prevents independent verification of every central claim in the paper, including Camlang's learnability, typological novelty, and the difficulty of the test set. A benchmark paper must make its central artifacts available to reviewers or provide a controlled access mechanism; otherwise, 'Camlang-CSQA-v0' is not a usable resource and the experimental results cannot be reproduced. This is a major reproducibility concern, not a presentation issue.
  4. [§5.3.2] The conclusion that 'most model successes stem from shallow lexical alignment' depends entirely on subjective human-verification labels ('correct and complete', 'incorrect and complete', 'correct and incomplete', 'incorrect and incomplete') assigned by the authors. The manuscript gives no operational definitions for completeness, no annotation rubric, and no inter-annotator reliability measure; it only states that disagreements were resolved by discussion and that 'only 9 required discussion.' Since the annotators know the gold answers and the model identities, confirmation bias is a real risk. The paper should provide annotation guidelines, a reliability study (e.g., Cohen's kappa on a held-out sample), and a pre-registered or at least fully specified procedure for mapping labels to SHV/MHV/LHV.
minor comments (6)
  1. [Abstract / §5.1.1] The abstract says 'Human experiments show these resources are sufficient', but the only comprehension experiment is a single participant. The plural 'experiments' is misleading; consider 'a human pilot' or report the full sample.
  2. [Table 6] The column heading 'ACC ↓' is ambiguous. It should be labelled 'Accuracy Drop (English − Camlang)' or similar. Also, the 'Time' column reports average inference time per instance, but for the human row this should be clarified as self-paced time per question.
  3. [§4.4, Figure 2] The word-count distribution is shown only for the question text, not for the answer options. Since the options were modified and a 'none of the above' option was added, reporting option lengths would help readers assess the difficulty shift.
  4. [§3.4.1] The ROUGE-based cross-annotator consistency scores are averaged over four annotators, but no standard deviations or significance tests are reported. Given the small number of annotators, these results are descriptive and should be presented as such.
  5. [Author Contributions] There is a typo: 'Camlan' should be 'Camlang'. Also, the contributions section credits individuals for specific experiments, which is useful, but the human participant's relationship to the annotation team should be made explicit in §5.1.1.
  6. [Appendix A] The WALS-style profile lists feature IDs and values, but without the full grammar and dictionary the mapping cannot be checked. If the resources remain withheld, the profile should at least be accompanied by a detailed description of how each feature value was determined.

Circularity Check

2 steps flagged · score 6.0 of 10

Human learnability evidence is self-referential: the 87% human baseline appears to be the Camlang test annotator, and the ROUGE learnability measure aggregates the language's own designers.

  1. self definitional [§5.1.1 (Baseline Systems and Implementation Details), Table 6; §4.3 (Annotation)]
    "Human Participant: A first-year undergraduate student in Linguistics at Oxford. … For human participant, to avoid leakage, we ask the annotator to complete the English questions after the Camlang questions. | we ask the core annotator, who participates in Camlang development, to translate them into Camlang, with reference to the grammar book and dictionary."

    The abstract and §5.2 use the human 87.23% Camlang score as the reference for 'far below human performance' and for the claim that humans can internalise grammatical systems from explicit resources. But §4.3 identifies the core annotator as a Camlang developer who translated the 47 test items; §5.1.1 refers to the human participant as 'the annotator'. On the paper's own wording, the human baseline is the person who wrote the translations and whose gold selections define correctness. Their 87% then measures the test constructor answering their own questions, not a naive adult learner acquiring Camlang from the grammar book and dictionary. The human-vs-LLM comparison is therefore not an independent test of human metalinguistic deduction; it is a self-referential yardstick.

  2. other [§3.2.3 (Iterative Refinement), §3.4.1 (Grammar Clarity and Practical Learnability), Table 2]
    "We assume that if the grammar–dictionary specification is sufficiently explicit and internally consistent, then the Camlang learners will converge on the same morphological choices and on similar clause-level linearization for a given English input. … We thus analyse the cross-annotator consistency as indirect evidence about the language learnability. | In each round, four annotators (linguistics undergraduates from Cambridge and Oxford who have experience in constructed languages and L2 acquisition) independently translated a set of English sentences or dialogues using the current grammar and"

    The cross-annotator ROUGE scores in Table 2 are presented as 'indirect evidence about the language learnability', but the annotators whose translations are scored are the same people who, in §3.2.3, iteratively designed, revised, and debugged Camlang and used translation divergences to patch the grammar. Their convergence reflects shared authorship and co-design conventions, not the behaviour of independent learners encountering the final grammar for the first time. Using this agreement to validate the grammar's clarity and learnability is circular: the designers' internal consistency is offered as evidence that the resource is learnable by outsiders.

full rationale

The LLM accuracy measurements themselves are not circular: the English control anchors the commonsense component, and model outputs were not authored by the paper's authors. However, the paper's central conclusion—that humans can reliably acquire Camlang from explicit rules while LLMs cannot—rests on two self-referential evidence sources. First, the sole human participant is described in §5.1.1 as 'the annotator', the same role that translated the test set and participated in Camlang development in §4.3; if so, the 87.23% human baseline is the benchmark constructor answering their own questions, not a naive learner. Second, the pre-registration-style learnability argument in §3.4.1 uses ROUGE agreement among the very annotators who co-designed and iteratively revised the grammar, so the 'learnability' evidence reduces to the designers agreeing with themselves. These two steps are load-bearing because the abstract's 'far below human performance at 87%' and 'humans can reliably internalise grammatical systems' are comparative claims that depend entirely on the human baseline. The model-vs-model comparisons (e.g., GPT-5 vs. GPT-4o) retain their empirical content, which is why the overall circularity score is 6 rather than higher; the human-centred conclusion is partially circular by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

No numerical parameters are fitted; the central comparison is empirical. The main quantitative inputs are author choices: test item selection, gold labels, verification labels, and the WALS feature annotation. These appear as domain assumptions and invented entities rather than free parameters.

assumptions (4)
  • domain assumption A combination of individually attested typological features suffices to guarantee that Camlang is 'unattested' and therefore leakage-free for LLMs.
    Invoked in Sections 3.1 and 3.3; the paper acknowledges in footnote 4 that an undocumented similar language may exist, and does not release the grammar to verify overlap.
  • domain assumption The authors' manual labels of model reasoning traces (correct/complete, etc.) reliably measure metalinguistic reasoning.
    Section 5.3.1; labels are assigned by the language's designers with no external or blinded protocol.
  • domain assumption One human participant's 87% EM approximates the human capability the benchmark is compared against.
    Section 5.1.1 lists a single first-year linguistics undergraduate; Section 7 says larger participant pools are future work.
  • domain assumption WALS feature overlap is an appropriate proxy for linguistic novelty.
    Section 3.4.2 uses WALS similarities; the feature values for Camlang are manually annotated by the authors, so the comparison is self-assessed.
invented entities (2)
  • Camlang
    purpose: A novel constructed language with grammar book and dictionary used as the testbed for metalinguistic reasoning.
    The full grammar and dictionary are withheld (Section 3.3); the only external handle is the WALS feature table in Appendix A, which is also author-assigned.
  • Camlang-CSQA-v0
    purpose: 47-item multiple-choice benchmark translating CommonsenseQA into Camlang.
    The dataset is not released; its gold labels are author-selected after post-hoc modifications (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang." pith.science (2026). https://pith.science/paper/XEUOOSZC

@misc{pith2026250900425,
  author       = {Pith},
  title        = {Pith review of: The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XEUOOSZC}},
  note         = {Machine review of arXiv:2509.00425}
}
read the original abstract

Large Language Models (LLMs) achieve gold-medal performance across many benchmarks, yet it remains unclear whether such success reflects genuine reasoning or pattern matching. From a cognitive science perspective, an informative test is whether models can master an unfamiliar language through explicit metalinguistic deductive learning, a paradigm where human learners can reliably internalise grammatical systems through metalinguistic reasoning. We address this question with Camlang, a novel constructed language that exhibits naturalistic yet unattested feature combinations. Camlang consists of two explicit resources, a grammar book and a bilingual dictionary, which mirror adult second-language learning via explicit grammar rules and lexical lookup, and enable us to disentangle errors in morpho-syntax, lexical semantics, and sentence-level reasoning. Human experiments show that these resources are sufficient for participants to acquire Camlang and successfully solve Camlang tasks. To operationalise evaluation, we adapt CommonsenseQA into Camlang, creating Camlang-CSQA-v0, the first task in a broader suite where solving questions requires applying grammar rules and lexical mappings. Experimental results show that GPT-5 achieves 98\% EM accuracy in English but only 47\% in Camlang, far below human performance at 87\%, while other state-of-the-art reasoning LLMs perform even worse. Human verification further reveals that most model successes stem from shallow lexical alignment while GPT-5 shows emerging metalinguistic awareness to a limited extent but not systematic grammatical mastery as humans. Camlang establishes a cognitively grounded evaluation paradigm that exposes fundamental gaps between current models and human metalinguistic competence.

Figures

Figures reproduced from arXiv: 2509.00425 by the authors.

Figure 1
Figure 1. Distribution of WALS features in documented languages (with selected languages [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Word count distribution of questions in English and Camlang. The [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Diagnosing CFG Interpretation in LLMs

    cs.AI 2026-04 unverdicted novelty 6.0 of 10

    LLMs maintain surface syntax for novel CFGs but fail to preserve semantics under recursion and branching, relying on keyword bootstrapping rather than pure symbolic reasoning.

Reference graph

Works this paper leans on

37 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [4]

    doi: 10.18653/v1/2024.findings-emnlp.916

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.916. URL https://aclanthology.org/2024.findings-emnlp.916/. Paola Beninc`a. The left periphery of medieval romance

    doi:10.18653/v1/2024.findings-emnlp.916 2024

  2. [8]

    Modeling: A novel dataset for testing linguistic reasoning in language models

    Nathan A Chi, Teodor Malchev, Riley Kong, Ryan A Chi, Lucas Huang, Ethan A Chi, R Thomas McCoy, and Dragomir Radev. Modeling: A novel dataset for testing linguistic reasoning in language models. arXiv preprint arXiv:2406.17038,

    arXiv

  3. [9]

    Herbert H Clark

    URL: https://www.nytimes.com/2023/03/08/opinion /noam-chomsky-chatgpt-ai.html . Herbert H Clark. Using language. Cambridge university press,

    work page 2023

  4. [14]

    The independence of phonology and morphology: The celtic mutations

    Antony D Green. The independence of phonology and morphology: The celtic mutations. Lingua, 116(11):1946–1985,

    work page 1946

  5. [18]

    Interview on ai and human-like understanding (nobel prize, dec 2024)

    Geoffrey Hinton. Interview on ai and human-like understanding (nobel prize, dec 2024). Nobel Prize Outreach Interview,

    work page 2024

  6. [19]

    Charles F Hockett and Charles D Hockett

    URL: https://www.nobelprize.org/prizes/phys ics/2024/hinton/interview/. Charles F Hockett and Charles D Hockett. The origin of speech. Scientific American, 203(3): 88–97,

    work page 2024

  7. [21]

    Back to School: Translation Using Grammar Books

    Jonathan Hus and Antonios Anastasopoulos. Back to school: Translation using grammar books. arXiv preprint arXiv:2410.15263,

    work page Pith review arXiv

  8. [22]

    doi: 10.18653/v1/2024.acl-long.787

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.787. URL https://aclanthology.org/2024.acl-long.787/. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254,

    doi:10.18653/v1/2024.acl-long.787 2024

Show all 37 references
  1. [27]

    Introducing O3 and O4-mini

    OpenAI. Introducing O3 and O4-mini. OpenAI Blog (Apr 2024),

    2024

  2. [28]

    Introducing GPT-5

    OpenAI. Introducing GPT-5. OpenAI Blog (Aug 2025),

    2025

  3. [29]

    Multi-LogiEval: Towards evaluating multi-step logical reasoning ability of large language models

    Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. Multi-LogiEval: Towards evaluating multi-step logical reasoning ability of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pr...

    2024

  4. [30]

    doi: 10.18653/v1/2024.emnlp-main.1160

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.1160. URL https://aclanthology.org/2024.emnlp-main.1160/. Martin J Pickering and Simon Garrod. An integrated theory of language production and comprehension. Behavioral and brain sciences, 36(4):329–347,

    2024 doi

  5. [31]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1602. URL https://aclanthology.o rg/2025.acl-long.1602/. Andrew Radford. English syntax: An introduction. Cambridge University Press,

    2025 doi

  6. [32]

    URL http://www.jstor.org/stable/44164110

    ISSN 00978507, 15350665. URL http://www.jstor.org/stable/44164110. Arthur S Reber. Implicit learning of artificial grammars. Journal of verbal learning and verbal behavior, 6(6):855–863,

  7. [33]

    Explain-then- process: Using grammar prompting to enhance grammatical acceptability judgments

    35 Working in Progress Russell Scheinberg, Ameeta Agrawal, Amber Shore, and So Young Lee. Explain-then- process: Using grammar prompting to enhance grammatical acceptability judgments. arXiv preprint arXiv:2506.02302,

    arXiv

  8. [34]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2019

  9. [36]

    Teaching large language models an unseen language on the fly

    Chen Zhang, Xiao Liu, Jiuheng Lin, and Yansong Feng. Teaching large language models an unseen language on the fly. arXiv preprint arXiv:2402.19167,

    arXiv

  10. [37]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.202. URL https://aclanthology.org/2025.acl-long.202/. 37 Working in Progress A Typological Features of Camlang To compare Camlang with other known languages, we manually annotate ...

    2025 doi

  11. [1905]

    doi: doi:10.1515/9783110833751-005

    ISBN 9783110833751. doi: doi:10.1515/9783110833751-005. URL https://doi.org/10.1515/9783110833751-0

    doi

  12. [1963]

    The FLORES evaluation datasets for low- resource machine translation: Nepali–English and Sinhala–English

    Francisco Guzm´an, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. The FLORES evaluation datasets for low- resource machine translation: Nepali–English and Sinhala–English. In Kentaro Inui, Jing Jiang, Vincent N...

    2019

  13. [1978]

    A corpus and cloze evaluation for deeper understanding of commonsense stories

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings o...

    2016

  14. [1992]

    Reduplication: a typological overview

    Colin Gorrie. Reduplication: a typological overview. Satyam Goyal and Soham Dan. Iolbench: Benchmarking llms on linguistic reasoning. arXiv preprint arXiv:2501.04249,

  15. [2002]

    URL https://www.science.org/doi/abs/10.1126/science.298.55 98.1569

    doi: 10.1126/sc ience.298.5598.1569. URL https://www.science.org/doi/abs/10.1126/science.298.55 98.1569. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of th...

  16. [2004]

    Bilingualism in development: Language, literacy, and cognition

    Ellen Bialystok. Bilingualism in development: Language, literacy, and cognition . Cambridge University Press, 2001a. 30 Working in Progress Ellen Bialystok. Metalinguistic aspects of bilingual processing. Annual review of applied linguistics, 21:169–181, 2001b. Ning Bian, Xian...

    2024

  17. [2005]

    doi: https://doi.org/10.1016/j.system.2004.12.006

    ISSN 0346-251X. doi: https://doi.org/10.1016/j.system.2004.12.006. URL https://www.scienc edirect.com/science/article/pii/S0346251X05000138. Nicholas Evans and Stephen C Levinson. The myth of language universals: Language diversity and its importance for cognitive science. Beh...

    2004 doi

  18. [2007]

    URL http://www.jstor.org/stable/27653511

    ISSN 00397857, 15730964. URL http://www.jstor.org/stable/27653511. Jennifer C White and Ryan Cotterell. Examining the inductive bias of neural language models with artificial languages. arXiv preprint arXiv:2106.01044,

  19. [2010]

    URL https: //aclanthology.org/C10-1044/

    Coling 2010 Organizing Committee. URL https: //aclanthology.org/C10-1044/. E Mark Gold. Language identification in the limit. Information and control, 10(5):447–474,

    2010

  20. [2016]

    doi: 10.18653/v1/N16-1098

    Association for Computational Linguistics. doi: 10.18653/v1/N16-1098. URL https://aclanthology.org/N16-1098/. Mariacristina Musso, Andrea Moro, Volkmar Glauche, Michel Rijntjes, J¨urgen Reichenbach, Christian B ¨uchel, and Cornelius Weiller. Broca’s area and the language insti...

    doi

  21. [2017]

    To ask LLMs about English grammaticality, prompt them in a different language

    Shabnam Behzad, Amir Zeldes, and Nathan Schneider. To ask LLMs about English grammaticality, prompt them in a different language. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 15622–156...

    2024

  22. [2018]

    Gary F Marcus, Sugumaran Vijayan, Shoba Bandi Rao, and Peter M Vishton

    URL https: //arxiv.org/abs/1804.08198. Gary F Marcus, Sugumaran Vijayan, Shoba Bandi Rao, and Peter M Vishton. Rule learning by seven-month-old infants. Science, 283(5398):77–80,

    arXiv

  23. [2019]

    doi: 10.18653/v1/D19-1632

    Association for Computational Linguistics. doi: 10.18653/v1/D19-1632. URL https://aclanthology.org/D19-1632/. Martin Haspelmath. The world atlas of language structures. Oxford University Press,

    doi

  24. [2020]

    doi: 10.18653/v1/2020.acl-main.421

    Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.421. URL https://aclanthology.org/2020.acl-main. 421/. Simone Balloccu, Patr´ıcia Schmidtov ´a, Mateusz Lango, and Ond ˇrej Duˇsek. Leak, cheat, repeat: Data contamination and evaluation malpractices in ...

    2020 doi

  25. [2021]

    UniSumm and SummZoo: Unified model and diverse benchmark for few-shot summarization

    Yulong Chen, Yang Liu, Ruochen Xu, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Yue Zhang. UniSumm and SummZoo: Unified model and diverse benchmark for few-shot summarization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting ...

    2023 arXiv

  26. [2022]

    Lingbench++: A linguistically-informed benchmark and reasoning framework for multi-step and cross-cultural inference with llms

    Da-Chen Lian, Ri-Sheng Huang, Pin-Er Chen, Chunki Lim, You-Kuan Lin, Guan-Yu Tseng, Zi-Cheng Yang, and Shu-Kai Hsieh. Lingbench++: A linguistically-informed benchmark and reasoning framework for multi-step and cross-cultural inference with llms. arXiv preprint arXiv:2507.16809,

    arXiv

  27. [2023]

    Evalu- ating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  28. [2024]

    Jump to better conclusions: Scan both left and right

    Jasmijn Bastings, Marco Baroni, Jason Weston, Kyunghyun Cho, and Douwe Kiela. Jump to better conclusions: Scan both left and right. arXiv preprint arXiv:1809.04640,

    arXiv

  29. [2025]

    How abilities in large lan- guage models are affected by supervised fine-tuning data composition

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large lan- guage models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.