Pith. sign in

REVIEW 2 major objections 4 minor 32 references

Instruction-Tuned Models Locally Reuse Human Syntax More Than Humans Do

T0 review · 2 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Instruction-tuned language models show greater raw syntactic overlap with the preceding human turn than the matched human response does, but the advantage reverses once the amount of available structure in the response is held constant.

desk verdict Careful measurement, overreaching title: the raw-overlap effect is real, but the paper's own conditioning reverses it, and the paper is honest about that tension. read the letter →

arxiv 2607.26015 v1 pith:DT2BDBOJ submitted 2026-07-28 cs.CL

Zandi Eberstadt This is my paper
classification cs.CL
keywords syntacticconvergenceinstructiontuningCFGrulereusestructuralpriminglanguagemodelssubstitutionparadigmopportunityalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether instruction-tuned language models converge on the syntax of the immediately preceding human turn, and whether they do so more than a human speaker would at the same position. Using a substitution paradigm in which model generations replace one speaker's turns in existing human dialogues, and measuring reuse of context-free grammar rules, the paper finds that all sixteen models reuse the true preceding turn's rules more than an unrelated prime, and that every instruction-tuned model shows greater raw rule-overlap than the human response it replaced. The central nuance is that this instruction-tuning advantage is a property of total natural-output alignment: when the amount of eligible syntactic structure in the response is held constant, instruction-tuned models show lower per-rule reuse odds than pretrained models, and in several architectures fall to the human baseline. The paper therefore distinguishes 'more overlap because more structure' from 'stronger tendency to reuse any given rule,' and argues the headline result belongs to the former category.

What carries the argument

The central machinery is CFG-rule overlap: each utterance is parsed into a constituency tree, the multiset of non-unary context-free productions is extracted, and binary overlap is coded for every eligible prime rule present in the response. A logistic mixed-effects model then contrasts three conditions at each of 1,901 matched positions—human response, model response, and model response against an unrelated prime—with rule frequency, prime size, and (in sensitivity analyses) target size as predictors. The target-size conditioning is the load-bearing piece: it separates raw overlap, which grows mechanically with the number of eligible rules a response contains, from per-rule reuse propensity

What would settle it

Recompute actual-prime CFG-rule overlap after truncating all human responses to 40 tokens or generating model responses without a length cap. If the instruction-tuned advantage over matched humans persists at matched target lengths, the structural-opportunity explanation is refuted; if it disappears or reverses, the headline claim is an artifact of the generation cap.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that instruction-tuned models locally reuse the CFG rules of the preceding human turn to a greater degree than the matched human respondent in raw output, with model-vs-human log-odds coefficients ranging from .398 to .796 across all eight instruction-tuned models, and that all eight matched pretrained/instruction-tuned architecture pairs show greater total actual-prime overlap after instruction tuning. At the same time, when target rule-set size is held constant, the instruction-minus-pretrained contrast reverses (β = −.505 full sample; β = −.423 in a central common-support sample), and the actual-versus-random increment—the boost from the true prime

Load-bearing premise

The central comparison assumes that a model generation capped at 40 tokens and the full-length human response it replaces are equally positioned measurements of syntactic convergence, so raw overlap differences can be read as behavioral rather than mechanical.

Editorial extensions

If this is right

  • If the structural-opportunity account is right, evaluations of 'convergence' or 'rapport' in chat models should always control for response length and available syntactic structure, not just raw overlap.
  • The universal low-frequency weighting of the actual-versus-random increment means a sensitivity to rare constructions is present in models before any instruction tuning; tuning weakens but does not eliminate it.
  • Because conditional per-rule reuse is lower under instruction tuning, high raw overlap and strong convergence propensity can diverge; papers measuring one cannot claim the other.
  • The consistent lexical and semantic advantage over humans suggests that models are biased toward the immediately preceding turn across multiple levels, but the split lexical result under tuning shows these levels are not governed by a single mechanism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment is to trace the same overlap metrics across intermediate checkpoints (pretraining, supervised fine-tuning, preference optimization); the paper notes this is unresolved, and locating where the structural expansion appears would test whether preference optimization implicitly rewards local mimicry.
  • Removing the 40-token generation cap could either amplify or eliminate the raw-alignment advantage; if longer generations dilute per-rule reuse, the human comparison would shift.
  • The 'amplification spiral' hypothesis about AI-induced delusional beliefs gains a concrete micro-mechanism: models preferentially echoing the immediately preceding syntax could increase perceived similarity without indicating shared understanding, which is a testable component of that broader account.
  • A cross-linguistic replication using translated or non-English dialogues would show whether the structural-opportunity account is language-general or an artifact of English short-dialogue corpora.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper uses a substitution paradigm in which model-generated turns replace one speaker's responses in human dialogues, and measures turn-adjacent reuse of CFG rules. Across sixteen Llama and Gemma models (pretrained and instruction-tuned), it reports three main results: (i) all models show greater overlap with the immediately preceding human turn than with a sampled unrelated human prime; (ii) every instruction-tuned model shows greater raw natural-output overlap with the actual prime than the matched human response it replaces, and all eight architecture pairs show greater raw overlap under instruction tuning; and (iii) conditioning on target rule-set size reverses or weakens the instruction-tuning advantage, and the per-model human contrasts are no longer uniformly positive. The paper also reports exploratory lexical/semantic similarity results and rule-frequency interactions. The analysis is transparent about the target-size confound, and the abstract explicitly notes the reversal under conditioning, but the title and 'Primary result II' frame the raw overlap as the central finding.

Significance. The paper is methodologically careful and unusually transparent for this literature. It uses matched substitution positions, a fixed random-prime seed shared across models, per-model and pooled GLMMs with confidence intervals, an exact matched-pair sign test, and a common-support sensitivity analysis. It also explicitly distinguishes total natural-output alignment from conditional reuse propensity and reports the reversal under target-size conditioning. These strengths make the unconditional effects credible and the paper a useful contribution to the study of LLM syntactic accommodation. However, the central claim as stated in the title—that instruction-tuned models 'reuse human syntax more than humans do'—is not supported by the conditional analysis, and the paper's own Table 2 shows several instruction-tuned models do not exceed humans once target structure is held constant. The contribution is real but requires reframing or additional matched analysis.

major comments (2)
  1. [§4.3, Table 2, Eq. (1)] The title and 'Primary result II' claim that instruction-tuned models 'reuse human syntax more than humans do.' The supporting evidence is the raw total-overlap contrast βA−H, which is a per-rule binary outcome; a target with more eligible structure has more opportunities to overlap, so this contrast conflates per-rule reuse propensity with target structural size. The paper's own sensitivity fit (Table 2, column βA−H|t) shows that, with target rule-set size held constant, three instruction-tuned models (Gemma-3-12B-IT .069 ns; Gemma-3-27B-IT −.081; Gemma-3-4B-IT −.023) no longer exceed matched humans. The unconditional effect is real and well estimated, but the behavioral interpretation 'reuse more than humans do' is not supported by the current evidence. Please rephrase the central claim to 'greater total natural-output overlap' or provide a target-size-matched human-model comparison th
  2. [§3.4, §4.3, Table 1] The reversal under conditioning is presented as evidence that instruction-tuned models produce more structure but have a lower 'tendency to reuse any given available rule.' However, target rule-set size is a property of the generated response—a post-treatment variable—so conditioning on it in a GLMM does not by itself identify a per-rule reuse propensity; it is a descriptive conditional association that may be affected by collider or overcontrol bias. The common-support analysis in Table 1 helps for the instruction-vs-pretrained comparison, but no common-support or matching analysis is provided for the model-vs-human contrasts in Table 2. Please either add a human-model comparison restricted to overlapping target-size regions, or weaken the causal-sounding 'tendency to reuse' language in the Discussion.
minor comments (4)
  1. [Title/Abstract] The title 'Locally Reuse Human Syntax More Than Humans Do' overstates the robust finding. The abstract's phrase 'greater natural-output overlap' is accurate; consider moving that qualifier into the title or adding an explicit caveat that the claim concerns raw overlap, not per-rule propensity.
  2. [§4.3] The sentence 'Pretrained models were closer to the human baseline as well as more heterogeneous; their coefficients ranged from −.030 to .240, and four were significantly positive while four were not distinguishable from zero' is clear, but the following sentence 'Pretrained models were closer to the human baseline' could be interpreted as all pretrained models being non-significant; specify that four were significant positive and four were not.
  3. [§3.4] The sensitivity fit is described as adding a centered target-size term as a main effect, but the pooled model also includes a tuning-by-target-size interaction and percentile-specific contrasts. Please provide the explicit model equation for the sensitivity and pooled specifications so readers can see exactly which terms are included.
  4. [Table 5] The common-support retention ranges from 33.2% to 95.5% across pair-by-tuning cells. Consider adding a sensitivity analysis restricted to positions retained for all cells, or discuss whether the imbalance affects the pooled common-support contrasts.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the measured overlaps are computed from fixed primes and generated targets, and the paper's own target-size conditioning provides the relevant robustness check.

full rationale

The derivation chain is self-contained. All contrasts (RQ1–RQ4) are computed from fixed human primes, sampled unrelated primes, and generated model/human targets; no outcome is used to define a predictor, no fitted value is relabeled as a prediction, and no load-bearing premise is sourced from a self-citation. The nearest issue is that the headline RQ2 comparison uses total natural-output overlap, which the paper itself decomposes: instruction-tuned outputs have larger target rule-set size, a smaller actual-versus-random increment, and a reversed pooled tuning contrast once centered log target size is conditioned upon (β=−.505 full sample, β=−.423 common support, Table 1). The per-model sensitivity fits in Table 2 and the explicit limitation in Section 6 ("Human responses are not subject to the same generation cap, making response structure an especially important part of the estimand") acknowledge that response-structure opportunity, not circularity, drives the raw-overlap difference. No equation reduces to its own inputs, and no self-citation is load-bearing. This is a statistical-confounding and interpretation concern, not a circularity in the derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No theoretical free parameters or invented entities are introduced. The listed design choices (eligibility threshold, generation cap, +1 offset) shape target-size distributions and therefore the raw-overlap comparisons; they are choices made by hand, not fitted constants.

free parameters (3)
  • Rule eligibility threshold = >1 occurrence in human corpus; unary rules excluded
    Defines the rule inventory and frequency scale; if the threshold changed, per-model contrasts and frequency analyses could shift (Section 3.2).
  • 40-token generation cap = 40
    Caps model response length, limiting structural opportunity; human responses are not capped, making the human-model comparison asymmetric (Section 3.1).
  • Target-size centering offset = 1
    log(|R(target)|+1) used in conditional models; the +1 offset affects conditional estimates for responses with zero eligible rules (Section 3.4).
assumptions (4)
  • domain assumption Constituency parse trees and CFG productions extracted by benepar are an adequate and comparable representation of syntactic structure for both human and LLM utterances.
    The entire dependent variable is CFG-rule reuse (§3.2); no parser validation is provided for model-generated text.
  • domain assumption A randomly sampled prime from a different conversation is a valid baseline isolating turn-adjacent syntactic convergence.
    The unrelated-prime condition is the RQ1 baseline and the source of the increment decomposition (§3.3).
  • domain assumption Matched substitution positions make model outputs and original human responses comparable at identical conversational positions.
    Human and model targets respond to the same prime, but generation cap and conditioning differ (§3.1); the raw comparison rests on this comparability.
  • domain assumption Rule eligibility (non-unary rules occurring more than once in the human corpus) does not distort frequency comparisons across human, pretrained, and instruction-tuned regimes.
    Eligibility affects the rule sets and frequency scale; frequencies are computed from human turns only (§3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instruction-Tuned Models Locally Reuse Human Syntax More Than Humans Do." pith.science (2026). https://pith.science/paper/DT2BDBOJ

@misc{pith2026260726015,
  author       = {Pith},
  title        = {Pith review of: Instruction-Tuned Models Locally Reuse Human Syntax More Than Humans Do},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DT2BDBOJ}},
  note         = {Machine review of arXiv:2607.26015}
}
read the original abstract

Syntactic convergence (the tendency of speakers to adapt in language towards the grammatical profiles of their interlocutors) is a well-documented feature of human dialogue widely considered to operate below conscious awareness. Whether large language models exhibit analogous syntactic convergence toward human users relative to human baselines and across a broad range of syntactic constructions remains an open question. Using substitution-paradigm data in which model generations replace one speaker's turns in pre-existing human dialogues, this study measures turn-adjacent reuse of context-free grammar (CFG) rules across sixteen open-weight Llama and Gemma models (1B-70B, pretrained and instruction-tuned) at 1,901 matched positions per model. Every model showed greater CFG-rule overlap with the preceding human turn than with a sampled unrelated human prime, and in every model this actual-versus-random difference was larger for lower-frequency rules. Each instruction-tuned model also showed greater natural-output overlap with the actual prime than the human response it replaced, and all eight matched architecture pairs exhibited greater actual-prime overlap after instruction tuning. However, relative to pretrained variants, instruction-tuned outputs overlapped more with unrelated primes, showed a smaller actual-versus-random increment, and had lower conditional rule-reuse odds once target rule-set size was held constant. In exploratory analyses, each model exhibited greater mean lexical and semantic similarity to the preceding turn than the matched human responses did. Instruction-tuned models additionally produced responses with greater mean semantic similarity than their pretrained counterparts in all eight architecture pairs, whereas the lexical similarity results were more heterogeneous.

Figures

Figures reproduced from arXiv: 2607.26015 by the authors.

Figure 1
Figure 1. Primary and pooled syntactic results. (a) Per-model actual-prime contrasts against the matched human response at centered mean rule frequency; error bars are 95% Wald intervals. (b) Pooled instruction-minus-pretraining coefficients under four estimands. Positive values favor instruction tuning. Conditional coefficients are evaluated at the target-size centering mean. Pretrained models were closer to the human baseli… view at source ↗
Figure 2
Figure 2. Human-minus-model similarity differences. Negative values mean that the model response [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [1]

    Cognitive psychology , volume=

    Syntactic persistence in language production , author=. Cognitive psychology , volume=. 1986 , publisher=

    1986

  2. [2]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  3. [3]

    2025 , eprint=

    Gemma 3 Technical Report , author =. 2025 , eprint=

    2025

  4. [4]

    , author=

    Becoming syntactic. , author=. Psychological review , volume=. 2006 , publisher=

    2006

  5. [5]

    7.2: Fixes for APIs and requirements , author=

    explosion/spaCy: v3. 7.2: Fixes for APIs and requirements , author=. Zenodo , year=

  6. [6]

    and Pickering, Martin J

    Branigan, Holly P. and Pickering, Martin J. and Cleland, Alexandra A. , title =. Cognition , year =

  7. [7]

    Journal of social and personal relationships , volume=

    Is actual similarity necessary for attraction? A meta-analysis of actual and perceived similarity , author=. Journal of social and personal relationships , volume=. 2008 , publisher=

    2008

  8. [8]

    and Bangerter, Adrian , title =

    Mayor, Eric and Bietti, Lucas M. and Bangerter, Adrian , title =. Cognitive Science , volume =. https://onlinelibrary.wiley.com/doi/pdf/10.1111/cogs.70106 , abstract =

    doi:10.1111/cogs.70106

Show all 32 references
  1. [9]

    and Arnett, Catherine and Chang, Tyler A

    Michaelov, James A. and Arnett, Catherine and Chang, Tyler A. and Bergen, Benjamin K. Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2...

    2023 doi

  2. [10]

    Cognition , volume=

    Alignment as a consequence of expectation adaptation: Syntactic priming is affected by the prime’s prediction error given both prior and recent experience , author=. Cognition , volume=. 2013 , publisher=

    2013

  3. [11]

    Perspectives on Psychological Science , volume=

    Artificial intelligence and the psychology of human connection , author=. Perspectives on Psychological Science , volume=. 2026 , publisher=

    2026

  4. [12]

    Contexts of Accommodation: Developments in Applied Sociolinguistics , editor =

    Giles, Howard and Coupland, Nikolas and Coupland, Justine , title =. Contexts of Accommodation: Developments in Applied Sociolinguistics , editor =. 1991 , pages =

    1991

  5. [13]

    and Garrod, Simon , title =

    Pickering, Martin J. and Garrod, Simon , title =. Behavioral and Brain Sciences , year =

  6. [14]

    , title =

    Reitter, David and Moore, Johanna D. , title =. Journal of Memory and Language , year =

  7. [15]

    Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

    Blevins, Terra and Schmalwieser, Susanne and Roth, Benjamin , title =. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , year =. doi:10.18653/v1/2026.eacl-long.34 , url =

    2026 doi

  8. [16]

    arXiv preprint arXiv:2605.29278 , year =

    Blevins, Terra , title =. arXiv preprint arXiv:2605.29278 , year =

    arXiv

  9. [17]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , year =

    Kandra, Florian and Demberg, Vera and Koller, Alexander , title =. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , year =. doi:10.18653/v1/2025.acl-short.68 , url =

    2025 doi

  10. [18]

    Findings of the Association for Computational Linguistics: ACL 2024 , year =

    Jumelet, Jaap and Zuidema, Willem and Sinclair, Arabella , title =. Findings of the Association for Computational Linguistics: ACL 2024 , year =. doi:10.18653/v1/2024.findings-acl.877 , url =

    2024 doi

  11. [19]

    Behavioral Sciences , year =

    Chen, Pengbo and Guan, Huining and Jeong, Eui Jun , title =. Behavioral Sciences , year =

  12. [20]

    Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , year =

    Li, Yanran and Su, Hui and Shen, Xiaoyu and Li, Wenjie and Cao, Ziqiang and Niu, Shuzi , title =. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , year =

  13. [21]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

    Kitaev, Nikita and Klein, Dan , title =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , year =

  14. [22]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year =

    Kitaev, Nikita and Cao, Steven and Klein, Dan , title =. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , year =

  15. [23]

    Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , year =

    Reimers, Nils and Gurevych, Iryna , title =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing , year =

    2019

  16. [24]

    Fitting Linear Mixed-Effects Models Using

    Bates, Douglas and M. Fitting Linear Mixed-Effects Models Using. Journal of Statistical Software , year =

  17. [25]

    Scandinavian Journal of Statistics , year =

    Holm, Sture , title =. Scandinavian Journal of Statistics , year =

  18. [26]

    Language and Linguistics Compass , volume=

    Structural priming: An experimental paradigm for mapping linguistic representations , author=. Language and Linguistics Compass , volume=. 2024 , publisher=

    2024

  19. [27]

    Journal of Memory and language , volume=

    The representation of verbs: Evidence from syntactic priming in language production , author=. Journal of Memory and language , volume=. 1998 , publisher=

    1998

  20. [28]

    Cognition , volume=

    Does verb bias modulate syntactic priming? , author=. Cognition , volume=. 2010 , publisher=

    2010

  21. [29]

    Cognition , volume=

    Syntactic priming of relative clause attachments: Persistence of structural configuration in sentence production , author=. Cognition , volume=. 2003 , publisher=

    2003

  22. [30]

    Proceedings of the workshop on cognitive modeling and computational linguistics , pages=

    Do large language models resemble humans in language use? , author=. Proceedings of the workshop on cognitive modeling and computational linguistics , pages=

  23. [31]

    Transactions of the Association for Computational Linguistics , volume=

    Structural persistence in language models: Priming as a window into abstract language representations , author=. Transactions of the Association for Computational Linguistics , volume=. 2022 , publisher=

    2022

  24. [32]

    NPP—Digital Psychiatry and Neuroscience , volume=

    Characterizing the spiral: potential mechanisms in AI-associated delusions , author=. NPP—Digital Psychiatry and Neuroscience , volume=. 2026 , publisher=

    2026

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.